https://wiki.jcihosting.com/api.php?action=feedcontributions&feedformat=atom&user=75.80.128.147JCWiki - User contributions [en]2026-04-21T02:02:03ZUser contributionsMediaWiki 1.42.3https://wiki.jcihosting.com/index.php?title=Private_IP_Mapping&diff=2246Private IP Mapping2020-12-09T19:24:42Z<p>75.80.128.147: /* Public IPs in Texas */</p>
<hr />
<div>= IP Mapping =<br />
<br />
== Public IP Mapping ==<br />
=== Castle (Redit) ===<br />
<pre><br />
69.55.224.0/24<br />
69.55.225.0/24<br />
69.55.226.0/24<br />
69.55.227.0/24<br />
69.55.228.0/24<br />
69.55.230.0/24<br />
69.55.232.0/24<br />
69.55.233.0/24<br />
69.55.234.0/24<br />
69.55.235.0/24<br />
69.55.236.0/24<br />
69.55.237.0/24<br />
69.55.238.0/24<br />
69.55.239.0/24<br />
<br />
<br />
</pre><br />
=== I2b ===<br />
<pre><br />
69.55.229.0/24<br />
69.55.231.0/24<br />
<br />
65.50.255.0/24 I2b Gig Link<br />
</pre><br />
<br />
=== Las Vegas ===<br />
<pre><br />
70.98.204.125-140<br />
netmask: 255.255.255.0<br />
gateway: 70.98.204.1<br />
dns: 70.99.204.249<br />
<br />
private: 172.16.204.127<br />
</pre><br />
<br />
<br />
== Private IP Mapping ==<br />
<br />
= Castle = <br />
<pre><br />
10.1.4.1 = nat <br />
10.1.4.3 = backup2<br />
10.1.4.4 = sun/console<br />
10.1.4.5 = mail<br />
10.1.4.8 = backup1<br />
10.1.4.10 = digi1<br />
10.1.4.11 = digi2? <br />
10.1.4.59 = virt9<br />
10.1.4.60-10.1.4.69 = virt10 - virt19<br />
10.1.4.90-10.1.4.91 = tempvirt1 - tempvirt2<br />
10.1.4.101-10.1.4.119 = jail1 - jail19<br />
10.1.4.151 = quar1<br />
10.1.4.160 = switch-p1a<br />
10.1.4.161 = switch-p1b<br />
10.1.4.162 = <br />
10.1.4.163 = switch-p3<br />
10.1.4.164 = switch-p4<br />
10.1.4.165 = <br />
10.1.4.166 = switch-p6<br />
10.1.4.167 = switch-p7<br />
10.1.4.169 = switch-p9<br />
10.1.4.160 = switch-p10<br />
10.1.4.161 = switch-p11<br />
10.1.4.162 = switch-p12<br />
10.1.4.163 = switch-p13<br />
10.1.4.164 = switch-p14<br />
10.1.4.165 = switch-p15<br />
10.1.4.166 = switch-p16<br />
10.1.4.167 = switch-p17<br />
10.1.4.168 = <br />
10.1.4.169 = <br />
10.1.4.170 = 3550<br />
10.1.4.171 = 3750<br />
10.1.4.172 = ASA<br />
10.1.4.201 = mx1<br />
10.1.4.202 = mx2<br />
10.1.4.203 = bwdb<br />
10.1.4.211 = DRAC virt11<br />
10.1.4.212 = DRAC virt12<br />
10.1.4.213 = DRAC virt13<br />
10.1.4.217 = DRAC virt17<br />
10.1.4.209 = DRAC virt19<br />
10.1.4.223 = gateway<br />
10.1.4.232 = DRAC jail2<br />
10.1.4.234 = DRAC jail4<br />
10.1.4.235 = RMM jail5<br />
10.1.4.237 = DRAC jail7<br />
10.1.4.238 = DRAC jail8<br />
10.1.4.239 = DRAC jail9<br />
</pre><br />
<br />
= i2b =<br />
<pre><br />
10.1.2.1 = nat2<br />
10.1.2.2 = firewall2<br />
10.1.2.3 = backup3<br />
10.1.2.4 = bwdb2<br />
10.1.2.5 = firewall3<br />
10.1.2.6 = backup4<br />
10.1.2.10 = digi3<br />
10.1.2.50 = switch-p20<br />
10.1.2.51 = switch-p21<br />
10.1.2.52 = switch-p22<br />
10.1.2.53 = switch-p23<br />
10.1.2.54 = switch-p24<br />
10.1.2.55 = switch-p25<br />
10.1.2.56 = switch-p26<br />
10.1.2.57 = switch-p27<br />
10.1.2.58 = switch-p28<br />
10.1.2.60 = switch-p30<br />
10.1.2.71 = ats-1<br />
10.1.2.72 = ats-2<br />
10.1.2.73 = ats-3<br />
10.1.2.74 = ats-4<br />
10.1.2.75 = ats-5<br />
10.1.2.76 = ats-6<br />
10.1.2.77 = ats-7<br />
10.1.2.78 = ats-8<br />
10.1.2.79 = ats-9<br />
10.1.2.103 = jail3<br />
</pre><br />
<br />
= Office =<br />
<pre><br />
10.1.6.1 = Motorola Gateway router<br />
</pre><br />
<br />
<br />
= Ground Water in Texas =<br />
== Public IPs in Texas ==<br />
<pre><br />
72.165.211.73 Gateway<br />
72.165.211.74 New Cisco 3750 Switch<br />
72.165.211.75 Power Switch<br />
72.165.211.76 NAT Box<br />
72.165.211.77 Cisco 3750 Switch<br />
72.165.211.78 New D-link Nat (old Digi addr) <br />
</pre><br />
<br />
== Private IPs in Texas ==<br />
<pre><br />
192.168.1.1 NAT Box<br />
192.168.1.2 Cisco 3750<br />
192.168.1.176 el1 Master Radio nm 255.255.255.248 gw 192.168.1.182<br />
192.168.1.177 el1 Slave Radio<br />
192.168.1.178 el1 Data Logger<br />
192.168.1.184/29 el2 Master Radio nm 255.255.255.248 gw 192.168.1.190<br />
192.168.1.185/29 el2 Slave Radio<br />
192.168.1.186/29 el2 Data Logger<br />
192.168.1.192 el3 Master Radio nm 255.255.255.248 gw 192.168.1.198<br />
192.168.1.193 el3 Slave Radio<br />
192.168.1.194 el3 Data Logger<br />
192.168.1.200/29 el4 Master Radio nm 255.255.255.248 gw 192.168.1.206<br />
192.168.1.201/29 el4 Slave Radio<br />
192.168.1.202/29 el4 Data Logger<br />
</pre></div>75.80.128.147https://wiki.jcihosting.com/index.php?title=Private_IP_Mapping&diff=2245Private IP Mapping2020-12-09T19:19:46Z<p>75.80.128.147: /* Public IPs in Texas */</p>
<hr />
<div>= IP Mapping =<br />
<br />
== Public IP Mapping ==<br />
=== Castle (Redit) ===<br />
<pre><br />
69.55.224.0/24<br />
69.55.225.0/24<br />
69.55.226.0/24<br />
69.55.227.0/24<br />
69.55.228.0/24<br />
69.55.230.0/24<br />
69.55.232.0/24<br />
69.55.233.0/24<br />
69.55.234.0/24<br />
69.55.235.0/24<br />
69.55.236.0/24<br />
69.55.237.0/24<br />
69.55.238.0/24<br />
69.55.239.0/24<br />
<br />
<br />
</pre><br />
=== I2b ===<br />
<pre><br />
69.55.229.0/24<br />
69.55.231.0/24<br />
<br />
65.50.255.0/24 I2b Gig Link<br />
</pre><br />
<br />
=== Las Vegas ===<br />
<pre><br />
70.98.204.125-140<br />
netmask: 255.255.255.0<br />
gateway: 70.98.204.1<br />
dns: 70.99.204.249<br />
<br />
private: 172.16.204.127<br />
</pre><br />
<br />
<br />
== Private IP Mapping ==<br />
<br />
= Castle = <br />
<pre><br />
10.1.4.1 = nat <br />
10.1.4.3 = backup2<br />
10.1.4.4 = sun/console<br />
10.1.4.5 = mail<br />
10.1.4.8 = backup1<br />
10.1.4.10 = digi1<br />
10.1.4.11 = digi2? <br />
10.1.4.59 = virt9<br />
10.1.4.60-10.1.4.69 = virt10 - virt19<br />
10.1.4.90-10.1.4.91 = tempvirt1 - tempvirt2<br />
10.1.4.101-10.1.4.119 = jail1 - jail19<br />
10.1.4.151 = quar1<br />
10.1.4.160 = switch-p1a<br />
10.1.4.161 = switch-p1b<br />
10.1.4.162 = <br />
10.1.4.163 = switch-p3<br />
10.1.4.164 = switch-p4<br />
10.1.4.165 = <br />
10.1.4.166 = switch-p6<br />
10.1.4.167 = switch-p7<br />
10.1.4.169 = switch-p9<br />
10.1.4.160 = switch-p10<br />
10.1.4.161 = switch-p11<br />
10.1.4.162 = switch-p12<br />
10.1.4.163 = switch-p13<br />
10.1.4.164 = switch-p14<br />
10.1.4.165 = switch-p15<br />
10.1.4.166 = switch-p16<br />
10.1.4.167 = switch-p17<br />
10.1.4.168 = <br />
10.1.4.169 = <br />
10.1.4.170 = 3550<br />
10.1.4.171 = 3750<br />
10.1.4.172 = ASA<br />
10.1.4.201 = mx1<br />
10.1.4.202 = mx2<br />
10.1.4.203 = bwdb<br />
10.1.4.211 = DRAC virt11<br />
10.1.4.212 = DRAC virt12<br />
10.1.4.213 = DRAC virt13<br />
10.1.4.217 = DRAC virt17<br />
10.1.4.209 = DRAC virt19<br />
10.1.4.223 = gateway<br />
10.1.4.232 = DRAC jail2<br />
10.1.4.234 = DRAC jail4<br />
10.1.4.235 = RMM jail5<br />
10.1.4.237 = DRAC jail7<br />
10.1.4.238 = DRAC jail8<br />
10.1.4.239 = DRAC jail9<br />
</pre><br />
<br />
= i2b =<br />
<pre><br />
10.1.2.1 = nat2<br />
10.1.2.2 = firewall2<br />
10.1.2.3 = backup3<br />
10.1.2.4 = bwdb2<br />
10.1.2.5 = firewall3<br />
10.1.2.6 = backup4<br />
10.1.2.10 = digi3<br />
10.1.2.50 = switch-p20<br />
10.1.2.51 = switch-p21<br />
10.1.2.52 = switch-p22<br />
10.1.2.53 = switch-p23<br />
10.1.2.54 = switch-p24<br />
10.1.2.55 = switch-p25<br />
10.1.2.56 = switch-p26<br />
10.1.2.57 = switch-p27<br />
10.1.2.58 = switch-p28<br />
10.1.2.60 = switch-p30<br />
10.1.2.71 = ats-1<br />
10.1.2.72 = ats-2<br />
10.1.2.73 = ats-3<br />
10.1.2.74 = ats-4<br />
10.1.2.75 = ats-5<br />
10.1.2.76 = ats-6<br />
10.1.2.77 = ats-7<br />
10.1.2.78 = ats-8<br />
10.1.2.79 = ats-9<br />
10.1.2.103 = jail3<br />
</pre><br />
<br />
= Office =<br />
<pre><br />
10.1.6.1 = Motorola Gateway router<br />
</pre><br />
<br />
<br />
= Ground Water in Texas =<br />
== Public IPs in Texas ==<br />
<pre><br />
72.165.211.73 Gateway<br />
72.165.211.74 New D-link Nat (old Digi addr)<br />
72.165.211.75 Power Switch<br />
72.165.211.76 NAT Box<br />
72.165.211.77 Cisco 3750 Switch<br />
72.165.211.78 New Cisco 3750 Switch<br />
</pre><br />
<br />
== Private IPs in Texas ==<br />
<pre><br />
192.168.1.1 NAT Box<br />
192.168.1.2 Cisco 3750<br />
192.168.1.176 el1 Master Radio nm 255.255.255.248 gw 192.168.1.182<br />
192.168.1.177 el1 Slave Radio<br />
192.168.1.178 el1 Data Logger<br />
192.168.1.184/29 el2 Master Radio nm 255.255.255.248 gw 192.168.1.190<br />
192.168.1.185/29 el2 Slave Radio<br />
192.168.1.186/29 el2 Data Logger<br />
192.168.1.192 el3 Master Radio nm 255.255.255.248 gw 192.168.1.198<br />
192.168.1.193 el3 Slave Radio<br />
192.168.1.194 el3 Data Logger<br />
192.168.1.200/29 el4 Master Radio nm 255.255.255.248 gw 192.168.1.206<br />
192.168.1.201/29 el4 Slave Radio<br />
192.168.1.202/29 el4 Data Logger<br />
</pre></div>75.80.128.147https://wiki.jcihosting.com/index.php?title=NetHere&diff=2243NetHere2020-09-01T23:49:42Z<p>75.80.128.147: /* nsrbl Disk Quota Warning */</p>
<hr />
<div>= NetHere =<br />
<br />
URL: http://toolbox.nethere.com<br />
<br />
<br><br />
Access to servers is restricted to admin-1.nethere.net. Access to admin-1.nethere.net is restricted to our office and nat.johncompanies.com.<br />
<br />
<br><br />
JCI Hosting acquired the web-hosting customers of NetHere. (Previously, NetHere had aquired customers from Simply Internet, and Z-net).<br />
Customer billing/info are being imported from NetHere to JCI database <br />
URL: https://secure.johncompanies.com/mgmt/index.html<br />
<br />
<br><br />
Nethere has several tools for management. These are restricted to only allow<br />
access from the office, or a few admin's homes.<br />
<br />
<br />
cacti [http://cacti.nethere.net]<br />
nagios[http://nagios.nethere.net]<br />
toolbox [http://toolbox.nethere.net]<br />
helpdesk [http://helpdesk.nethere.net]<br />
support FAQ [http://www.nethere.net/website/support]<br />
knowledge base [http://support.nethere.net/kb]<br />
phpmyadmin [https://phpmyadmin.nethere.net]<br />
<br />
<br />
== Virtual Hosts ==<br />
<br />
<pre><br />
<br />
66.63.129.2 - New HP DL360<br />
fpweb-2.nethere.net<br />
unixweb-6.nethere.net<br />
mta-db.mail.nethere.net<br />
scan-1.mail.nethere.net<br />
<br />
66.63.129.4- vmware9.eng<br />
mysqldb-1.webhost.nethere.net<br />
<br />
66.63.129.91<br />
lists-1.nethere.net<br />
sb-2.nethere.net<br />
mx-1.nethere.net- spooling server<br />
mailx-1.nethere.net- web server outbound mail server<br />
<br />
66.63.129.101 (c.host.nethere.net)<br />
ahi.nethere.net- ldap<br />
mailbox-4<br />
mta-2.mail.nethere.net<br />
relay-2.mail.nethere.net<br />
unixweb-8.nethere.net<br />
<br />
66.63.129.102 (vm2.eng.nethere.net)<br />
koi.nethere.net- off- realserver<br />
prov-1.nethere.net- cp.nethere.net, cp.znet.net, cp.simplyweb.net<br />
scribe.nethere.net- prov.nethere.net, scribe.nethere.net<br />
tetra.nethere.net- nethere.com, znet.com, simplyweb.com<br />
web-2.inboxhq.net- helpdesk<br />
webmail-1.nethere.net<br />
<br />
66.63.129.103<br />
andromeda.nethere.net (ns2.nethere.net)<br />
nscache-2.nethere.net<br />
nsrbl-2.nethere.net<br />
mailbox-2.nethere.net<br />
scan-2.mail.nethere.net<br />
<br />
66.63.129.104<br />
eel.nethere.net- ldap<br />
mta-1.mail.nethere.net<br />
nscache-1.nethere.net<br />
nsrbl-1.nethere.net<br />
phoenix.nethere.net (ns1.nethere.net)<br />
relay-1.mail.nethere.net<br />
unixweb-2.nethere.net<br />
<br />
66.63.129.105<br />
home-1.nethere.net<br />
ntweb-11.nethere.net<br />
ntweb-6.nethere.net<br />
cart32.nethere.net<br />
(web-3.schedulecafe.com)<br />
<br />
66.63.129.106 6499MB/16381 18.32GB/268.25 free<br />
admin-1.nethere.net<br />
admin-2.nethere.net<br />
pike.nethere.net- cacti, nagios- off<br />
shark.nethere.net- ldap<br />
ntdb-1.nethere.net<br />
winrestore64<br />
<br />
207.167.93.106- vmware6.eng 4156MB/8185MB 45.94GB/267GB free<br />
ntweb-2.nethere.net<br />
unixweb-12.nethere.net (no web sites on server)<br />
mailbox-1.nethere.net<br />
<br />
207.167.93.108- vmware8.eng<br />
mailx-2.nethere.net<br />
mx-2.nethere.net<br />
<br />
207.167.93.110- vmware10.eng<br />
unixweb-11.nethere.net<br />
</pre><br />
<br />
== Hardware Hosts ==<br />
<br />
<pre><br />
Cab 6-08<br />
<br />
siron-3 (off)<br />
siron-2<br />
siron-1<br />
106<br />
104<br />
sndg-br-1<br />
sndg-br-2<br />
<br />
mailbox-2.nethere.net (off - virtualized)<br />
mailbox-3.nethere.net <br />
mailbox-1.nethere.net (off - virtualized)<br />
<br />
scan-1 (OFF)<br />
<br />
scan-2 <br />
<br />
129.2<br />
102<br />
101<br />
<br />
105<br />
103<br />
<br />
Backup-2<br />
Backup-1<br />
<br />
</pre><br />
<br />
<pre><br />
Cab 5-02<br />
<br />
ntweb-3<br />
unixweb-7<br />
<br />
unixweb-3<br />
unixweb-2 (now virtualized)<br />
unixweb-10<br />
<br />
ntdb-2 or 3?<br />
<br />
vmware6.eng<br />
<br />
<br />
</pre><br />
<br />
== Admin Access ==<br />
<br />
Access to the Admin Systems are protected by firewalls and<br />
application filters based on IP addresses.<br />
<br />
=== Main Firewall ===<br />
The Main firewall is on sndg-cr-1. The main firewall is a Cisco 3750.<br />
<br />
==== IP Access List Editing ====<br />
<br />
<pre><br />
1. enable<br />
<br />
2. show ip access-lists access-list-name <br />
<br />
show ip access-list BLOCKED<br />
<br />
3. configure terminal<br />
<br />
4. ip access-list resequence access-list-name starting-sequence-number increment<br />
<br />
5. ip access-list {standard | extended} access-list-name<br />
<br />
ip access-list extended BLOCKED <br />
<br />
6. sequence-number permit source source-wildcard<br />
sequence-number deny protocol source source-wildcard destination destination-wildcard [precedence precedence] [tos tos] [log] [time-range time-range-name] [fragments]<br />
no sequence-number <br />
<br />
7. end<br />
<br />
8. show ip access-lists access-list-name<br />
</pre><br />
<br />
=== Host Firewall ===<br />
On the servers, there is often a firewall such as ipfw, ipf, or pf.<br />
<br />
==== Toolbox ====<br />
<br />
From root@admin-1 connect to scribe <br />
ssh scribe<br />
<br />
Stop the scrolling of messages to the screen<br />
/etc/rc.d/syslogd stop<br />
<br />
Add the new IP address to the pf config.<br />
vi /etc/pf.conf<br />
<br />
Reload the firewall rules<br />
pfctl -F all -f /etc/pf.conf<br />
<br />
Add the new IP address to the apache config.<br />
cd /nethere/conf/apache/conf/vhosts<br />
co -l prov.nethere.net scribe.nethere.net.common<br />
vi prov.nethere.net<br />
vi scribe.nethere.net.common<br />
ci -u prov.nethere.net scribe.nethere.net.common<br />
<br />
Reload the apache config<br />
apachectl restart<br />
<br />
Restart the syslog daemon<br />
/etc/rc.d/syslogd start<br />
<br />
==== Nagios ====<br />
<br />
From root@admin-1 connect to pike <br />
ssh nagios<br />
<br />
Add the new IP address to the apache config.<br />
cd /nethere/conf/apache/conf/vhosts<br />
co -l nagios.nethere.net cacti.nethere.net mrtg.nethere.net<br />
vi nagios.nethere.net cacti.nethere.net mrtg.nethere.net<br />
ci -u nagios.nethere.net cact.nethere.net mrtg.nethere.net<br />
<br />
Reload the apache config<br />
apachectl restart<br />
<br />
=== Application Filter ===<br />
In the applications, there is often a restriction in the host application<br />
configuration file.<br />
vi /usr/local/etc/apache/conf/vhosts/<host>.conf<br />
<br />
Restart the application<br />
apachect restart<br />
<br />
<br />
== NetHere/zNET/simplyweb Cert replacement ==<br />
<br />
<br />
If you wish to use the old CSR, Global Sign keeps the old<br />
CSR, so all you have to replace is the Cert.<br />
<br />
<li>To generate a new csr for each<br />
<pre><br />
openssl req -nodes -newkey rsa:2048 -keyout wild.nethere.key -out wild.nethere.csr<br />
openssl req -nodes -newkey rsa:2048 -keyout wild.simplyweb.key -out wild.simplyweb.csr<br />
openssl req -nodes -newkey rsa:2048 -keyout wild.znet.key -out wild.znet.csr<br />
</pre><br />
<br />
<li>Get a wild card cert for each company<br />
(nethere.net, simplyweb.net. znet.net)<br />
<br> <br />
https://www.globalsign.com/en/<br />
<br><br />
<br />
<li>Update certs, keys, and/or pem files on the following servers.<br />
<br />
<pre><br />
webmail-1.mail (webmail for nh, si, and zn) /nethere/conf/apache/pki/ssl.crt<br />
prov-1.mail (cp for nh, si, and zn) /nethere/conf/apache/pki/ssl.crt<br />
mta-1 and mta-2 (smtp) admin-2:/dist/files/nhmta/nethere/conf/postfix/nh/ssl<br />
/nethere/conf/postfix/nh/ssl<br />
/nethere/conf/postfix/si/ssl<br />
/nethere/conf/postfix/zn/ssl<br />
mailbox-1, mailbox-2, and mailbox-3 (pop3 and imap) (courier-imap)<br />
/nethere/conf/courier-imap/nh<br />
/nethere/conf/courier-imap/si<br />
/nethere/conf/courier-imap/zn<br />
(smtp)<br />
</pre><br />
<br />
</pre><br />
<br />
== Provisioning ==<br />
<br />
<br />
<br />
=== Webservers ===<br />
<br />
<br />
OS: FreeBSD 4.11-RELEASE-p10 #23<br />
<br />
<br />
==== General ====<br />
<br />
<pre><br />
unixweb-1 through 8<br />
<br />
All have the same configuration except for the following exceptions:<br />
<br />
unixweb-4: ZN Front Page server (depreciated)<br />
unixweb-5: ZN Front Page, Miva Merchant server<br />
unixweb-6: SI shared counter server (for SI sites): /www/lucy.inetworld.net/htdocs/cgi-bin/Count.cgi <br />
<br />
Server Software Installed<br />
<br />
Apache<br />
# httpd -v<br />
Server version: Apache/2.0.55<br />
Server built: Apr 5 2006 17:04:01<br />
PHP<br />
# /usr/local/bin/php4 -v<br />
PHP 4.4.2 (cgi-fcgi) (built: Mar 2 2006 09:31:57)<br />
<br />
Miva Merchant (unixweb-5)<br />
MySQL<br />
# mysqladmin version -u root -p<br />
Server version 4.0.25<br />
<br />
Apache configurations are in the following directories:<br />
<br />
/nethere/conf/apache/conf/nvhosts (name based hosting); naming convention is www.domain.tld<br />
/nethere/conf/apache/conf/vhosts (ip based,SSL); naming convention is www.domain.tld (IP information) and www.domain.tld.common (general site information)<br />
<br />
NetHere specific scripts are located in:<br />
/nethere/sbin<br />
<br />
Checking Server/site status via mod_status apache module:<br />
<br />
# apachectl stop<br />
# /usr/local/etc/rc.d/httpd.sh start-status<br />
<br />
URL: http://unixweb-#/status (replace # with the unixeweb number)<br />
** NOTE: After viewing status, do the following, otherwise other users can also view status by using .htaccess.<br />
# apachectl stop<br />
# apachectl startssl<br />
</pre><br />
<br />
==== Provisioning ====<br />
Provisioning new sites:<br />
<br />
*** All website provisioning is done via NH script (webadd) on sawfish (admin-1.nethere.net) ***<br />
# /nethere/sbin/webadd -h for usage<br />
<br />
Note: We've discontinued new website provisioning on all servers except for unixweb-6, unless otherwise noted for domain<br />
<br />
<br />
==== FTP Password Adjustment ====<br />
<br />
===== Unix Server =====<br />
To change an FTP password:<br />
<br />
On the customers server as root do<br />
<br />
passwd <username><br />
<br />
(The password prompt will not echo entry)<br />
<br />
===== Windows Server =====<br />
<br />
To change an FTP password:<br />
<br />
<pre><br />
Find the users login (ie aa5014) in the provisioning tool.<br />
Use Remote Desktop to get into the server.<br />
On Desktop find FTP application Icon.<br />
Use that tool to update the user's password.<br />
</pre><br />
<br />
==== Domain Aliasing ====<br />
<br />
For domain aliasing:<br />
<br />
1) Edit the /named/named.master on phoenix (ns1.nethere.net)<br />
a) Checkout the file; open with vi<br />
# co -l named.master<br />
# vi named.master<br />
b) find the domain that's serving as the master domain, then, following the general format of the file to add the domain aliases under the master domain entry, using the master domain zone file<br />
i.e. <br />
--cut--<br />
zone "domain.tld" { type master; file "master/domain.tld"; };<br />
zone "aliased_domain.tld" { type master; file "master/master_domain.tld"; }; <br />
--cut--<br />
c) save the file, then check-in:<br />
# ci -u named.master < /dev/null<br />
d) update the zone records, using the makefile in /named; check for errors<br />
# make new-zone<br />
# tail /var/log/named<br />
**Note: some domains are IP based virtually hosted (i.e. SSL certificate). This requires a modification to the standard aliasing procedure. If this is the case, do the following<br />
e) Create a new zone file called "domain.tld-alias" in /named/master, copy current domain.tld zone file to the domain.tld-alias file, and adjust the "website" records, removing whatever IP based information is there, and replacing with the appropriate $INCLUDE name-based host template.<br />
e.g.<br />
tropicalshade.net:<br />
--cut--<br />
;; ntweb-4<br />
@ IN A 66.63.136.4<br />
www IN A 66.63.136.4<br />
--cut--<br />
tropicalshade.net-alias<br />
--cut--<br />
;; website<br />
$INCLUDE master/nvhost.ntweb-4<br />
--cut--<br />
f) Adjust the named.master zone file in /named accordingly:<br />
i.e. <br />
--cut--<br />
zone "aliased_domain.tld" { type master; file "master/master_domain.tld-alias"; }; <br />
--cut--<br />
<br />
<br />
2) Edit the Apache configuration for the domain on the web server the master domain is hosted on<br />
a) Script is /nethere/sbin/webalias<br />
# /nethere/sbin/webalias -h (for usage)<br />
e.g. # /nethere/sbin/webalias -d www.originaldomain.tld -a www.domainalias.tld<br />
<br />
<br />
==== CGI Scripting ====<br />
<br />
For security, we've implemented suexec on all sites.<br />
All CGI *must* be placed in the cgi-bin/ (and/or cgibin/ directory for unixweb-4, 5) directory<br />
<br />
CGI permissions break down as follows:<br />
<br />
Ownership: user=username, group=webuser<br />
Permission: cgi-bin/ = 755; files = 755<br />
<br />
suexec errors can be found here:<br />
/www/default/logs/suexec_log<br />
<br />
cgi errors for a domain can be found in the main apache error logs for the domain:<br />
/www/www.domain.tld/logs/error_log<br />
<br />
More information on CGI scripting can be found here:<br />
http://httpd.apache.org/docs/1.3/howto/cgi.html<br />
<br />
<br />
==== Formmail Provisioning ====<br />
<br />
<br />
Formmail is provisioned via NH script (newformmail) on server domain is hosted on.<br />
# /nethere/sbin/newformmail -h for usage<br />
<br />
Usage: newformmail [-h] [-d domain] [-i ip_address] [-e "email1 email2"]<br />
<br />
<br />
==== Disabling a site ====<br />
<br />
<br />
To disable/enable a website, there is a script (webstatus) located on each webserver:<br />
/nethere/sbin/webstatus -h for usage<br />
<br />
1) Disable: <br />
# /nethere/sbin/webstatus -r -d -s www.domain.tld<br />
<br />
2) Enable:<br />
# /nethere/sbin/webstatus -r -e -s www.domain.tld<br />
<br />
==== Stats Provisioning ====<br />
<br />
We utilize webalizer for statistics.<br />
<br />
Stats are provisioned via NH scripts on server domain is hosted on.<br />
<br />
1) Provision stats for the domain (webaddstats_unix):<br />
# /nethere/sbin/webaddstats_unix -d www.domain.tld<br />
<br />
2) Run stats for the domain (runstats_unix):<br />
# /nethere/sbin/runstats_unix www.domain.tld<br />
<br />
More information on Webalizer can be found here:<br />
http://www.mrunix.net/webalizer<br />
README: ftp://ftp.mrunix.net/pub/webalizer/README<br />
<br />
<br />
==== Removing a site ====<br />
<br />
1) Site removal is primarily done via a NH script (webdel) on the webserver the domain is hosted on:<br />
# /nethere/sbin/webdel -h for usage<br />
# /nethere/sbin/webdel -r -d www.tldomain.com <br />
<br />
2) Update DNS accordingly (ns1.nethere.net)<br />
a) Checkout /named/named.master<br />
# co -l named.master<br />
b) Remove line containing domain<br />
c) Checkin /named/named.master<br />
# ci -u named.master < /dev/null<br />
d) Update zones<br />
# make new-zone<br />
e) Move zone files from /named/master to /named/archive<br />
# mv domain<br />
<br />
==== starter removal ==== <br />
<br />
All starter sites on home-1.nethere.net<br />
<br />
1) Removal primarily done via a NH script (webdel_home) on home-1<br />
# nethere/sbin/webdel_home -h for usage<br />
i.e. webdel_home -p zn -s startername <== for removing znet<br />
<br />
<br />
<br />
==== Disk Quota increases ====<br />
<br />
<br />
Quotas are increased by using the "edquota" command:<br />
# edquota username<br />
Note that quotas are in KB, so a conversion is necessary. Also note that the "soft" quota is 1 MB less than the "hard" quota.<br />
<br />
The calculations are as follows:<br />
"soft" = ( quota(in MB) - 1 ) x 1024<br />
"hard" = quota(in MB) x 1024<br />
e.g. for a 500 MB quota:<br />
soft => (500 - 1) x 1024 = 510976<br />
hard => 500 x 1024 = 512000<br />
<br />
To check a quota:<br />
# quota -v username<br />
<br />
You can also use the prototype users:<br />
<br />
# edquota -p quota100 username (100 MB)<br />
# edquota -p quota500 username (500 MB)<br />
# edquota -p quota1000 username (1000 MB)<br />
<br />
==== Granting of shell - Enterprise packages ====<br />
<br />
<br />
Shell is granted by the "chsh" command. By default, C shell (csh) is used:<br />
# chsh -s shell_needed username<br />
e.g. # chsh -s csh username<br />
<br />
*Note: customers must provide us with the static IP address(es) they will be connecting from. The IP address(es) must be added to the configuration on admin-1.<br />
1) Modify the corresponding rules files on admin-1:<br />
/dist/files/nhweb/etc/ipf.rules.fxp0 , .em0 (RCS controlled)<br />
2) Push updates to servers<br />
# cd /dist/rdist<br />
# gmake nhweb<br />
# gmake nhweb-update<br />
3) Reload the rules on the corresponding server that needs access to<br />
# ipf -Fa -f /etc/ipf.rules<br />
<br />
<br />
==== SSL certificate installs ====<br />
<br />
<br />
<br />
1) Whois the site for information on Registrant, email - to be used for SSL generation<br />
# whois domain.tld<br />
<br />
2) Create SSL self signed certificate, get CSR for customer to sign<br />
a) SSL generation is done by script: /nethere/conf/apache/pki/newsslcert.sh <br />
b) Copy CSR for customer<br />
<br />
3) Create/update Apache configuration files via NH script (webadd_ssl)<br />
# /nethere/sbin/webadd_ssl -h for usage<br />
a) webadd_ssl [-h] [-d domain] [-s ssl_domain] [-n]<br />
# webadd_ssl -d www.domain.tld -s www.domain.tld <br />
<br />
4) Update DNS zone with new IP address - done on ns1.nethere.net<br />
a) Check out DNS zone file in /named/master<br />
# co -l domain.tld<br />
b) Adjust A records:<br />
--cut--<br />
;; unixweb-## << enter the unixweb server number for ease of ID<br />
@ IN A vhost_ip_address<br />
www IN A vhost_ip_address<br />
--cut-- <br />
c) Check in DNS zone file<br />
# ci -u domain.tld < /dev/null<br />
d) Reload zone file<br />
# rndc reload domain.tld<br />
e) check /var/log/named for errors<br />
<br />
5) Update DNS PTR record for IP address - done on ns1.nethere.net<br />
Note: Assuming IP address a.b.c.d<br />
a) Check out in-addr.arpa zone for IP address in /named/master<br />
# co -l a.b.c <br />
b) Follow format for PTR records --cut--<br />
d IN PTR www.domain.tld.<br />
--cut--<br />
c) Check in PTR zone file # ci -u a.b.c < /dev/null<br />
d) Reload PTR zone <br />
# rndc reload c.b.a.in-addr.arpa<br />
e) check /var/log/named for errors<br />
<br />
<br />
==== Updating SSL certificate ====<br />
<br />
<br />
1) cd to /nethere/conf/apache/pki/ssl.crt on server site is hosted on<br />
<br />
2) Check out (RCS) www.domain.tld.crt file<br />
# co -l www.domain.tld.crt<br />
<br />
3) Edit the file, remove old certificate, paste in new certificate<br />
<br />
4) Check in (RCS) the www.domain.tld.crt file<br />
# ci -u www.domain.tld.crt < /dev/null<br />
<br />
5) Check, restart Apache<br />
# apachectl configtest<br />
# apachectl stop<br />
# apachectl startssl<br />
<br />
6) Verify httpd started: <br />
# ps auxw | grep httpd<br />
a) If no processes, will need to revert back to old SSL cert, and restart apache. Check logs for errors<br />
# view /www/default/logs/ssl_engine.log<br />
b) Check for "Unable to configure RSA server private key" and "key values mismatch" entries - this means a bad SSL certificate<br />
<br />
<br />
<br />
==== Password protection ====<br />
<br />
<br />
Http (simple) password protection is governed by the Apache configuration for the domain<br />
<br />
1) Create userdb, users file in the domain root directory (/www/www.domain.tld):<br />
# mkdir userdb<br />
# cd userdb<br />
# htpasswd -bc users username password<br />
<br />
2) Check out Apache config for www.domain.tld in /nethere/conf/apache/conf/<vhosts,nvhosts><br />
# co -l www.domain.tld(.common) <br />
<br />
3) Edit Apache configuration, add the following lines within the VirtualHost container <br />
--cut--<br />
<Directory "/www/www.domain.tld/dir_to_be_protected"><br />
AuthType Basic<br />
AuthName "www.domain.tld/dir_to_be_protected authentication"<br />
AuthUserFile /www/www.domain.tld/userdb/users <br />
<Limit GET POST><br />
require valid-user<br />
</Limit><br />
</Directory><br />
--cut--<br />
<br />
4) Check in Apache config<br />
# ci -u www.domain.tld(.common) < /dev/null <br />
<br />
5) Restart Apache<br />
# apachectl configtest<br />
# apachectl restart<br />
<br />
More on http (simple) password protection can be found here:<br />
http://httpd.apache.org/docs/1.3/howto/auth.html#basic<br />
<br />
<br />
==== .htaccess ====<br />
<br />
<br />
Used if customers want control of certain Apache directives (i.e. Authentication, etc.)<br />
<br />
1) Check out Apache config for www.domain.tld in /nethere/conf/apache/conf/<vhosts,nvhosts><br />
# co -l www.domain.tld(.common) <br />
<br />
2) Add the AllowOverride directive in the <Directory> section, under the PHP FCGIWrapper<br />
i.e.<br />
<Directory "/www/www.domain.tld/htdocs"><br />
FCGIWrapper /www/www.domain.tld/htdocs/cgi-bin/php4 .php<br />
AllowOverride AuthConfig FileInfo Indexes Limit<br />
</Directory><br />
<br />
3) Check in Apache config<br />
# ci -u www.domain.tld(.common) < /dev/null <br />
<br />
4) Restart Apache<br />
# apachectl configtest<br />
# apachectl restart<br />
<br />
<br />
More info on Allow Override can be found here: http://httpd.apache.org/docs/1.3/mod/core.html#allowoverride<br />
<br />
<br />
==== Domain re-provisioning ====<br />
<br />
<br />
## NOTE: Following is for domain being provisioned on same server (i.e. just being renamed)<br />
<br />
1) Adjust DNS - ns1.nethere.net<br />
a) Rename the DNS zone file to the new domain, remove the old DNS zone files from /named/master and /named/master/RCS<br />
b) Check in the new DNS zone file<br />
c) Edit /named/named.master - replace the old domain with the new one, alias as necessary<br />
d) Reload DNS zones - make new-zone in /named<br />
<br />
2) Adjust Apache config - server domain is hosted on<br />
a) Find current config file(s) in /nethere/conf/apache/conf/<vhosts,nvhosts><br />
b) Replace old domain name entries with new domain name, alias as necessary<br />
c) Save as new_domain.tld<br />
d) Remove old_domain.tld(.common), RCS/old_domain.tld(.common)<br />
e) Check in new_domain.tld via RCS:<br />
# ci -u new_domain.tld < /dev/null<br />
f) Check out Apache include configuration file via RCS /nethere/conf/apache/conf/<nvhosts.conf,vhosts.conf> <br />
g) Edit nvhosts.conf or vhosts.conf, replace old_domain.tld entries with new_domain.tld<br />
h) Check in via RCS /nethere/conf/apche/conf/<nvhosts.conf,vhosts.conf><br />
<br />
3) Rename directory for new domain<br />
# cd /www<br />
# mv www.old_domain.tld www.new_domain.tld<br />
<br />
4) Adjust PHP stub files/configuration<br />
a) Adjust php.ini file in /www/www.domain.tld/(php4,php5)<br />
b) Adjust PHP stub files in /www/www.domain.tld/htdocs/cgi-bin/(php4,php5) <br />
*note: need to chflags to "noschg" for /www/www.domain.tld/htdocs/cgi-bin/(php4,php5) before being able to update the stub files<br />
after adjusting, be sure to chflags schg /www/www.domain.tld/htdocs/cgi-bin/(php4,php5)<br />
<br />
5) Edit password file (vipw), replace old_domain.tld entries with new_domain.tld<br />
# vipw<br />
<br />
6) Restart Apache<br />
# apachectl configtest<br />
# apachectl restart<br />
<br />
## NOTE: Following is for domain being re-provisioned on new server<br />
<br />
1) Run NH script "webadd" on sawfish to provision domain on new server, *DO NOT* reload DNS, use same user/pass as before.<br />
2) After customer has uploaded site to new server and gives the ok do the following:<br />
a) Update DNS zone for domain.tld; reload zone for domain.tld<br />
b) Wait 48 hours, then remove the site off the old server<br />
# /nethere/sbin/webdel<br />
<br />
==== FTP space provisioning ====<br />
<br />
<br />
Generally FTP sites are provisioned on the server that hosts the main website. In the case of NT based FTP sites, we usually provision them on the server with the most space available.<br />
<br />
1) Provision site on server<br />
a) Done via NH script (webadd_ftp): webadd_ftp [-h] [-d domain] [-u username] [-p password] [-n]<br />
# /nethere/sbin/webadd_ftp -h <- for usage<br />
EX: for ftp.domain.tld:<br />
# /nethere/sbin/webadd_ftp -d ftp.domain.tld -u username -p password<br />
(leaving off the -n will restart the proftpd process)<br />
b) Note: The host IP address will be given when the provisioning is completed, use that ip (a.b.c.d) for DNS entries<br />
c) Note: If this is an existing customer on the server, you'll need to increase the quota manually by 100 MB for the customer, see the section on quota increases for more info.<br />
<br />
2) Adjust DNS for domain on ns1.nethere.net<br />
a) Checkout zone for domain<br />
b) Add ftp host entry for domain, adjust serial<br />
EX for domain.tld:<br />
--cut--<br />
ftp IN A a.b.c.d<br />
--cut--<br />
c) Check in zone for domain<br />
d) Reload zone<br />
# rndc reload domain.tld<br />
3) Adjust PTR record for domain, adjust serial<br />
a) Checkout zone for a.b.c<br />
b) Add record for domnain:<br />
--cut--<br />
d IN PTR ftp.domain.tld.<br />
--cut--<br />
c) Check in zone for a.b.c<br />
d) Reload zone<br />
# rndc reload c.b.a.in-addr.arpa<br />
4) Check for DNS errors<br />
a) tail /var/log/namedb<br />
<br />
<br />
<br />
==== SiteBuilder provisioning ====<br />
<br />
<br />
unixweb-7.nethere.net<br />
<br />
*Must re-provision site on unixweb-7.nethere.net (if not already done)<br />
<br />
1) Log in to SB admin:<br />
http://sitebuilder.nethere.net/admin<br />
a) username: root<br />
<br />
2) Add site to SB config<br />
a) Click on Site Management -> Add regular<br />
* Alias is website username: i.e. aa####<br />
* Check the "Active" box <br />
* Plan is "BasePlan" <br />
* Password same as website<br />
b) Click on "Publish Properties<br />
* Check "Allow publishing"<br />
* Site host name: www.domain.tld<br />
* FTP host: unixweb-7.nethere.net<br />
* FTP login/password: same as site user/pass<br />
* FTP working directory: leave blank<br />
c) Click Apply<br />
<br />
<br />
<br />
<br />
==== Name Servers ====<br />
<br />
<br />
OS: FreeBSD<br />
<br />
<br />
==== General ====<br />
<br />
<br />
ns1.nethere.net - Primary name servers for DNS zone records<br />
/named - contains the files that have all DNS domain zone entries (named.master, named.slave, named.acl) plus Makefile for distributing DNS records<br />
/named/master - contains all the domain zone files for which we are authorative for, as well as IP address (PTR records)<br />
<br />
ns2.nethere.net - Secondary (slave) name server for DNS zone records<br />
<br />
nsrbl-1.nethere.net - RBL (Realtime Blackhole List) DNS server<br />
/named/rbldns/cache - contains the files for domains that we specifically allow or deny<br />
<br />
nscache-1,2 - caching name servers<br />
<br />
<br />
<br />
==== Adding DNS website entries ====<br />
<br />
<br />
1) Create a DNS zone file for domain.tld via NH script (zoneadd_vhost)<br />
# /nethere/sbin/zoneadd_vhost -h (for usage)<br />
<br />
<br />
<br />
==== Adding DNS IP entries ====<br />
<br />
<br />
<br />
1) Create the forward and reverse DNS records via NH script (zone_generate) for a netblock, will create /tmp/customer.forward and /tmp/customer.reverse files to be read<br />
a) /nethere/sbin/zone_generate -h for usage<br />
i.e. for netblock a.b.c, starting IP d, ending IP z<br />
# zone_generate -n a.b.c -b d -e z -p customer<br />
<br />
2) Checkout, edit the IP in-addr.arpa zone file, reload the zone<br />
a) # co -l a.b.c<br />
b) Seach for the nearest netblock area for the domain, follow format for customer info, read in the /tmp/abbrev.rdns file accordingly, increase Serial for zone in YYYYMMDD## format<br />
e.g.<br />
--cut--<br />
;;;;<br />
;; 66.63.152.232/30 (255.255.255.252)<br />
;; Description: First Choice Home Improvement<br />
;; Contact: Shannon Hill <firstchoicehi@hotmail.com>, (858) 277-5351<br />
;; Location: AR-1, Serial3/0/18:0<br />
;;;;<br />
232 IN PTR firstchoice-net.access.nethere.net.<br />
233 IN PTR firstchoice-gw.access.nethere.net.<br />
234 IN PTR firstchoice-2.access.nethere.net.<br />
235 IN PTR firstchoice-bcast.access.nethere.net.<br />
--cut--<br />
c) # ci -u a.b.c < /dev/null<br />
d) # rndc reload c.b.a.in-addr.arpa<br />
e) verify loading of zone: # tail /var/log/named<br />
3) Checkout, edit the forward DNS zone file, reload the zone<br />
a) # co -l access.nethere.net<br />
b) Search for the nearest neblock area for the domain, follow format for customer, read in the /tmp/abbrev.fdns file accordingly, increase Serial for zone in YYYYMMDD## format<br />
e.g.<br />
--cut--<br />
;; 66.63.152.232/30 (255.255.255.252)<br />
firstchoice-net IN A 66.63.152.232<br />
firstchoice-gw IN A 66.63.152.233<br />
firstchoice-2 IN A 66.63.152.234<br />
firstchoice-bcast IN A 66.63.152.235<br />
--cut--<br />
c) # ci -u access.nethere.net < /dev/null<br />
d) # rndc reload access.nethere.net <br />
e) verify loading of zone: # tail /var/log/named<br />
<br />
<br />
<br />
<br />
==== Unblocking RBLd IP addresses ====<br />
<br />
There are two primary reasons why we add customers to the allow relay list:<br />
a) They have a static IP address (i.e. DSL, T1) and wish to use our mail servers<br />
b) They've been blocked by one of our subscribed blackhole lists, however, have patched their machine and are no longer open to relay.<br />
<br />
To do this on nsrbl-1.nethere.net:<br />
<br />
1) Checkout the allow.relays.nethere.net file located in /named/rbldns/cache<br />
<br />
2) Edit the file, and add the IP address in the following format:<br />
--cut--<br />
a.b.c.d YYYYMMDD hostname reason for listing<br />
--cut--<br />
<br />
You can also add subnets via '/' notation for relay<br />
--cut--<br />
a.b.c.d/28 YYYYMMDD hostname reason for listing<br />
--cut--<br />
<br />
3) Check in the file<br />
<br />
4) Updates to the rbldns zone are done automatically on the hour, every hour, so no need to do anything else. <br />
<br />
Note:<br />
<br />
For "permanent" (i.e. customer static IP addresses), add the IP address in the "## permanent allowed relay (i.e. customer w/ static IP)" section<br />
For "temporary" (i.e. blackhole listed IP addresses), add the IP address in the "## temporary" section<br />
<br />
Also, for temporary IP addresses, we need to send the note to the requester detailing the following:<br />
a) If the IP address relays Spam/UCE/Viruses through us, it is to be removed permanently.<br />
b) The customer must follow the steps listed on the blackhole list the IP was listed on to get removed.<br />
<br />
<br />
<br />
==== Flushing DNS cache for a domain ====<br />
<br />
<br />
Due usually to a bad zone or excessively long TTL for a domain, the cache for it will need to be flushed.<br />
<br />
On nscache-1:<br />
# /usr/local/sbin/rndc flushname domain.tld<br />
<br />
If cache is still corrupted (i.e. zone lookups either fail or are incorrect), need to stop and restart the caching server:<br />
<br />
# /etc/init.d/local.named stop<br />
# /etc/init.d/local.named start<br />
<br />
==== SiteBuilder ====<br />
http://sitebuilder.nethere.net/<br />
-URL used to test sitebuilder<br />
<br />
http://sitebuilder.nethere.net/Login<br />
-URL for control panel. <br />
<p>This is where the customer also logs in to manage<br />
their web site and also where you log in to administer sitebuilder.<br />
Sitebuilder is hosted on sb-2.nethere.net. The site is designed/built on<br />
this server and published to unixweb-7.nethere.net. This is the only server<br />
that can host a sitebuilder web site. The admin log in to manage<br />
sitebuilder is:<br />
<br />
Username: admin<br />
Password: N3tH3r31!<br />
<br />
==== Cart32 ====<br />
cart32 information:<br />
<br />
Cart32 is hosted on ntweb-6.nethere.net. That is where all of the<br />
configuration files are located. The location of the ini file is<br />
D:\websites\Cart32cgi/cart32.ini. This is where ip restrictions to admin<br />
panel are set, password can be reset for admin, time limit restriction<br />
reset, etc.<br />
<br />
There are four customers that still use cart32:<br />
<br />
<pre><br />
https://www.cart.simplyweb.net/lab400/cart/c32web.exe<br />
https://www.cart.simplyweb.net/nutragenics/cart/c32web.exe<br />
https://www.cart.simplyweb.net/retrogen/cart/c32web.exe<br />
https://www.cart.simplyweb.net/stonesculptorssupplies/cart/c32web.exe<br />
</pre><br />
<br />
The client codes are lab400, nutragenics, retrogen, and<br />
stonesculptorssupplies. You can reset their passwords through the admin<br />
panel.<br />
<br />
To administer cart32:<br />
<br />
Control Panel: https://www.cart.simplyweb.net/cart/c32web.exe/Admin<br />
Username: administrator<br />
Password: N3tH3r31!<br />
Cart Admin Password: N3tH3r31!<br />
<br />
=== Webmail ===<br />
<br />
<br />
<br />
==== Removing webmail filters ====<br />
<br />
All webmail filters are stored in the MySQL database on webmail-1.<br />
<br />
Simple SQL commands:<br />
<br />
* Choosing a DB<br />
mysql> use db_name;<br />
<br />
* Showing tables in a DB (must be using a DB)<br />
mysql> show tables;<br />
<br />
* Describing a table's fields<br />
mysql> describe table_name;<br />
<br />
<br />
For NetHere, the database is horde_nh.<br />
For Simply, the database is horde_si.<br />
For zNET, the database is horde_zn.<br />
<br />
1) Log to MySQL as root, using the normal root password<br />
a) # mysql -u root -p<br />
2) Select the appropriate database<br />
a) mysql> use horde_nh<br />
3) Delete webmail filters.<br />
a) mysql> delete from horde_prefs where pref_uid='<username>' and pref_name='filters';<br />
Replace <username> with the users login.<br />
<br />
<br />
==== Removing Address book entries ====<br />
<br />
1) Log in<br />
2) Select appropriate db<br />
3) Find object_id, delete<br />
a) mysql> select object_id from turba_objects where object_email='email@domain.tld';<br />
b) mysql> delete from turba_objects where object_id='object_id_obtained_previously';<br />
<br />
<br />
==== Repairing webmail address book ====<br />
<br />
<br />
1) Export (via webmail, if possible) the address book to a csv file.<br />
<br />
2) Delete the Turba objects in the mySQL database in the turba_objects table that is assigned to the customer.<br />
mysql> delete from turba_objects where owner_id='username';<br />
<br />
3) Import the saved address book via webmail.<br />
<br />
<br />
==== Changing webmail From Information ====<br />
<br />
1) Login as the customer (http://webmail.nethere.net)<br />
<br />
2) Click on Options<br />
<br />
3) Click on Personal Information<br />
<br />
4) Select either default identity or a new one and click on Edit Your Identities<br />
<br />
=== MySQL ===<br />
<br />
<br />
All unixweb-## boxes have MySQL processes running on them. DB provisioning is done on the same server as which hosts the domain.<br />
For Windows 2000 MySQL provisioning, all DBs are placed on ntdb-2.nethere.net<br />
<br />
<br />
==== MySQL DB provisioning ====<br />
<br />
<br />
Use https://phpmyadmin.nethere.net <br />
User: root<br />
PW: <root db pw><br />
<br />
1) Select the server that the DB needs to be hosted on, typically the same server that the website is hosted on.<br />
2) Create the DB<br />
a) Under "MySQL" - "Create new database" - the DB name is typically the website username; click "Create"<br />
3) Create the User, set permissions for user on DB<br />
a) Click "Home" (upper left), then click "Privileges" (Under "MySQL")<br />
b) Click "Add a new User" make sure "Any host" selected, "User name" is typically the website username, "Password" is typically the website password. !LEAVE "Global Privileges" settings deselected! Click "Go" (bottom right corner) when finished.<br />
c) Under "Database-specific privileges" be sure to add the user database<br />
d) Select DB, then for "Privileges:" make sure the following are checked:<br />
Select, Insert, Update, Delete, Create, Alter, Index, Drop, Create Temporary Tables, Lock Tables, References<br />
<br />
<br />
=== List Server ===<br />
<br />
lists-1.nethere.net<br />
<br />
<br />
==== Mail List provisioning ====<br />
<br />
<br />
1. Go to the /mailman/bin directory:<br />
# cd /mailman/bin<br />
<br />
2. Create the list on lists-1:<br />
# ./newlist -q \<br />
listname@lists.domain.com \<br />
mailman-owner@lists.domain.com \<br />
password <br />
<br />
3. Configure the list using default settings:<br />
# ./config_list -i /mailman/data/defaultlist.cfg listname<br />
<br />
*** IF A NEW DOMAIN FOLLOW INSTRUCTIONS BELOW ***<br />
<br />
NOTE: All files in /etc/mail are controlled by RCS<br />
<br />
4. Go to the /etc/mail directory:<br />
# cd /etc/mail<br />
<br />
5. Add the list domain to the mailertable file:<br />
lists.domain.com mailman:lists.domain.com<br />
<br />
6. Add the list domain to the relay-domains file.<br />
<br />
7. Add the list domain to the virtuserdomain file.<br />
<br />
8. Add the following entries to the virtusertable file:<br />
mailman@lists.domain.com <customer_email_address><br />
mailman-owner@lists.domain.com mailman@lists.domain.com<br />
<br />
9. Run make to rebuild all files:<br />
# make<br />
<br />
10. Run make to restart sendmail:<br />
# make restart<br />
<br />
<br />
=== Backup Server ===<br />
<br />
backup-1 -> odd numberd unix<br />
backup-2 -> even numbered unix<br />
<br />
<br />
On Saturdays, we do a level 0 dump<br />
the rest are differential<br />
<br />
Restoration fees are basically $25 per day restored for email, $150 for website from cancelled archive, $25 a day for website <br />
<br />
Restoration is based on customer request. In order to restore a site/mailbox fully, you must start restoration from the previous level 0 backup and continue until the day after the day requested, since backups are performed in the morning.<br />
<br />
General rule is to leave the gzipped file available for 24 hours, after that remove it.<br />
<br />
<br />
==== Site Restoral ====<br />
<br />
<br />
If the site was removed by the web_del script, then so long as it is within 30 days since removal, the gzipped site will be available within /www/archive/www.domain.tld-date_removed.tar.gz<br />
Extract with tar:<br />
# tar -zxvpf www.domain.tld-date_removed.tar.gz<br />
<br />
If the site needs to be restored from backup, here are the steps<br />
<br />
1) To determine where backups are stored, go to /nethere/conf/backup<br />
<br />
2) grep for the server within the directory to determine the backup directory used<br />
i.e. <br />
# grep unixweb-1 *<br />
<br />
4) Go to /backup1/unix/unixweb-1.nethere.net<br />
a) you'll see a bunch of directories, named <date>-<dump_level><br />
<br />
5) Since the backups happen in the morning, typically you'll need to go to the day before, restore<br />
e.g. for 11/10<br />
a) cd 20041109-3<br />
b) restore -if and whatever the filesystem is named<br />
(for interactive: add files/dirs, extract, then use 1 for volume #)<br />
i.e.<br />
restore> add <path_to_dir/files><br />
restore> extract<br />
(for volume #: 1)<br />
(Set owner: y)<br />
<br />
6) After you've restored the files, just use scp as root on sawfish to copy them off of backup-# and then onto unixweb-#<br />
a) (on sawfish - two step process)<br />
# scp backup-1:/tmp/<restored_file> /tmp<br />
# scp /tmp/<restored_files> unixweb-1:/tmp<br />
<br />
7) Cleanup any restored files off of backup-# server and sawfish<br />
<br />
<br />
With the newly restored files, following are the steps necessary to restore a site:<br />
<br />
1) Restore www.domain.tld directory to /www<br />
# mv /www/archive/www.domain.tld /www<br />
<br />
2) Restore configuration files<br />
a) Apache: mv /www/archive/nethere/conf/apache/(n)vhosts/www.domain.tld /nethere/conf/apache/conf/(n)vhosts/<br />
b) Webalizer: mv /www/archive/nethere/conf/webalizer/unix/www.domain.tld /nethere/conf/webalizer/unix<br />
<br />
3) Add user to /etc/master.passwd:<br />
a) vipw<br />
(go to end of file, read in the master.passwd file from /www/www.domain.tld)<br />
:r /www/www.domain.tld/master.passwd<br />
b) Remove /www/www.domain.tld/master.passwd<br />
<br />
4) Change flags on cgi-bin/ directories:<br />
# chflags sunlnk /www/www.domain.tld/htdocs/cgi-bin/<br />
# chflags schg /www/www.domain.tld/htdocs/cgi-bin/php*<br />
<br />
5) Add configuration file entry to nvhosts/vhosts.conf<br />
# co -l (n)vhosts.conf<br />
# vi (n)vhosts.conf<br />
# ci -u (n)vhosts.conf<br />
<br />
6) Check apache, restart; verify apache<br />
# apachectl configtest<br />
# apachectl graceful<br />
# <br />
<br />
7) Cleanup remaining restored files<br />
# rm -r /www/archive/www<br />
# rm -r /www/archive/nethere<br />
<br />
<br />
8) Enter zone into DNS on ns1.nethere.net<br />
a) zone file<br />
# mv /named/archive/domain.tld /named/master<br />
# mv /named/archive/domain.tld,v /named/master/RCS<br />
b) named.master file <br />
# co -l /named/named.master<br />
(add zone)<br />
# ci -u /named/named.master<br />
# cd /named; make new-zone<br />
# tail /var/log/named (look for errors)<br />
*************************************************<br />
(12:40:47 PM) Henry Chan: restore is now available that is compatible with the 4.4bsd format<br />
(12:41:04 PM) Henry Chan: to restore, use the following command: restore -c -i -f path_to_archive<br />
(12:41:08 PM) Henry Chan: the "-c" is what does it<br />
(12:41:29 PM) Henry Chan: (only applies to backup-1-new... doesn't work on backup-2 or backup-4)<br />
<br />
<br />
==== Email Restoral ====<br />
<pre><br />
Storage path prefixes:<br />
/nfs/1 is sndg-netapp-1 (on backup-4 - /backup/hosts/2/e0.sndg-netapp-1.nethere.net)<br />
/nfs/2 is sndg-netapp-2 (on backup-3 - /backup/hosts/2/e0.sndg-netapp-2.nethere.net)<br />
/nfs/3 is sndg-netapp-3 (on backup-2 - /backup/hosts/2/e0.sndg-netapp-3.nethere.net)<br />
/nfs/4 is sndg-netapp-1 (on backup-2 - /backup/hosts/2/e0.sndg-netapp-1.nethere.net)<br />
/nfs/5 is sndg-netapp-2 (on backup-1 - /backup/hosts/2/e0.sndg-netapp-2.nethere.net)<br />
/nfs/6 is sndg-netapp-3 (on backup-2 - /backup/hosts/2/e0.sndg-netapp-3.nethere.net)<br />
<br />
Looking up a mailbox's Storage Path:<br />
You will need to know which directory and NFS server the customer's mail is <br />
stored on. Use the Provisioning Tool to get the Storage Path (find the customer's mail<br />
account, then click on the Engineering sub-tab under the Email tab). You should end up<br />
with something like "/nfs/3/nh/h/t/htchan/Maildir". Match the prefix of the storage<br />
directory with an NFS server (see prefixes above).<br />
<br />
<br />
<br />
1) If restoring a recently deleted mail account:<br />
a) Look for the archive on mailbox-1 in /nfs/archive/mail/{platform}/{username}.{date}-{PID}.tar.gz. <br />
If it doesn't exist, it's been too long and the only way to get email back is to restore from backup.<br />
b) Make sure the account is re-created in Provisioning Tool and look up the storage path <br />
(see "Looking up a mailbox's Storage Path" above).<br />
c) Extract the archive to a temporary directory:<br />
Sample command:<br />
cd /tmp; tar xzvpf /nfs/archive/mail/nh/zella.20090902-7508.tar.gz<br />
Sample output:<br />
nfs/2/nh/z/e/zella/<br />
nfs/2/nh/z/e/zella/Maildir/<br />
nfs/2/nh/z/e/zella/Maildir/tmp/<br />
nfs/2/nh/z/e/zella/Maildir/new/<br />
nfs/2/nh/z/e/zella/Maildir/cur/<br />
nfs/2/nh/z/e/zella/Maildir/maildirsize<br />
nfs/2/nh/z/e/zella/Maildir/.Trash/<br />
nfs/2/nh/z/e/zella/Maildir/.Trash/tmp/<br />
nfs/2/nh/z/e/zella/Maildir/.Trash/new/<br />
nfs/2/nh/z/e/zella/Maildir/.Trash/cur/<br />
nfs/2/nh/z/e/zella/Maildir/.Trash/maildirfolder<br />
nfs/2/nh/z/e/zella/Maildir/.Drafts/<br />
nfs/2/nh/z/e/zella/Maildir/.Drafts/tmp/<br />
nfs/2/nh/z/e/zella/Maildir/.Drafts/new/<br />
nfs/2/nh/z/e/zella/Maildir/.Drafts/cur/<br />
nfs/2/nh/z/e/zella/Maildir/.Drafts/maildirfolder<br />
nfs/2/nh/z/e/zella/Maildir/.Sent Items/<br />
nfs/2/nh/z/e/zella/Maildir/.Sent Items/tmp/<br />
nfs/2/nh/z/e/zella/Maildir/.Sent Items/new/<br />
nfs/2/nh/z/e/zella/Maildir/.Sent Items/cur/<br />
nfs/2/nh/z/e/zella/Maildir/.Sent Items/maildirfolder<br />
nfs/2/nh/z/e/zella/Maildir/courierpop3dsizelist<br />
Note:<br />
In the above output, notice that the Maildir is located in "nfs/2/nh/z/e/zella", it will be used in the next step.<br />
d) Copy the Maildir files to the new mailbox storage path using tar (tar does a better job at preserving <br />
things like symbolic links and permissions than cp or mv):<br />
Sample command:<br />
cd /tmp/nfs/2/nh/z/e/zella; tar cf - Maildir | (cd /nfs/3/nh/z/e/zella; tar xvpf -)<br />
Output:<br />
You should list the files as it is copied. Basically, similar to what you <br />
saw in step "c" above but without the "nfs/2/nh/z/e/zella" prefix.<br />
Note:<br />
The command format is basically:<br />
cd /tmp/{restored-maildir-path-see-note-in-step-c}; tar cf - Maildir | (cd {new-storage-path-without-Maildir}; tar xvpf -)<br />
e) Clean up temporary directory:<br />
Sample command:<br />
cd /tmp; rm -rf nfs<br />
<br />
2) If restoring a zfs mailbox (from the last week) to a particular date's backup:<br />
a) login to the correct sndg-netapp-[1/2/3]-new<br />
b) get the files from the correct zfs snapshot<br />
Sample commands:<br />
cd /tank0/mail/.zfs/snapshot/20180215-0/nh/c/h/christineat<br />
ls -l<br />
cd to the desired directory and copy the files over to the users mailbox<br />
cp -p /tank0/mail/.zfs/20180215-0/nh/c/h/christineat/Maildir/cur /tank0/mail/nh/c/h/christineat/Maildir/cur<br />
<br />
3) If restoring a mailbox to a particular date's backup:<br />
a) Look up the storage path to the customer's mailbox and determine which backup server the <br />
dump file is on (see "Looking up a mailbox's Storage Path" and "Storage path prefixes" above).<br />
b) Restore the level 0 and any incremental backups (in order) to /tmp on the backup server:<br />
Sample commands:<br />
cd /tmp<br />
restore4x -if /backup/hosts/2/e0.sndg-netapp-2.nethere.net/20090912-0/mail.dump<br />
chflags -R 0 /tmp<br />
restore4x -if /backup/hosts/2/e0.sndg-netapp-2.nethere.net/20090913-1/mail.dump<br />
chflags -R 0 /tmp<br />
...skipped repetitive stuff here...<br />
restore4x -if /backup/hosts/2/e0.sndg-netapp-2.nethere.net/20090918-6/mail.dump<br />
chflags -R 0 /tmp<br />
Note:<br />
After each restore, we need to recursively remove all flags from /tmp to eliminate <br />
the immutable flag that gets set on the files (this happens only on NetApp dumps... who knows why).<br />
c) Create a new tar of the Maildir directory so that it can be copied to mailbox-1 for further processing:<br />
Sample command:<br />
cd /tmp/nh/z/e/zella; tar cf /tmp/archive.tar Maildir<br />
d) Clean up /tmp:<br />
Sample command:<br />
rm -rf /tmp/nh<br />
Note:<br />
Depending on the platform, it might be /tmp/nh, /tmp/si, or /tmp/zn.<br />
e) Copy the /tmp/archive.tar file to /tmp on mailbox-1.<br />
f) On mailbox-1, extract the Maildir archive on top of what they already have:<br />
Sample command:<br />
cd /nfs/2/nh/z/e/zella; tar xvpf /tmp/archive.tar<br />
g) Remove /tmp/archive.tar.<br />
</pre><br />
<br />
==== Beginning of old instructions ====<br />
Use the same techniques as site restoration, with the following exceptions:<br />
<br />
1) Since everything is in maildir format, you have to reassmble things in /tmp/<username>, tar it up, and then restore it on any of the mailbox machines<br />
a) using the following syntax: <br />
# tar -zcpf /tmp/username.tgz yyyymmdd-#/platform yyyymmdd-#/platform yyyymmdd-#/platform<br />
i.e.<br />
# tar -zcpf /tmp/username.tgz 20050915-5/nh 20050914-4/nh 20050910-0/nh<br />
<br />
2) In regards to mail spools, you'll need to use the chflags command to adjust the flags on the files. For whatever reason, they are stored with a system immutable flag<br />
a) Do chflags -R noschg <dir> on the restored directory before copying the files with sawfish otherwise, the files cannot be deleted:<br />
i.e. <br />
# chflags -R noschg /backup1/unix/sndg-netapp-1-e2b.nethere.net/20050105-4/<br />
b) After tar/gzipping, remove the restored directories<br />
i.e<br />
# rm -r /backup1/unix/sndg-netapp-1-e2b.nethere.net/20050105-4/nh<br />
<br />
3) Can restore quickly with script on mailbox-1 (or mailbox-2) using the NH script "restoremail"<br />
a) # /nethere/sbin/restoremail -h for usage<br />
i.e. <br />
# restoremail -p nh -f nhusername.tgz -n 2 -u nhusername<br />
*** End of old instructions ***<br />
<br />
<br />
<br />
* Backup locations *<br />
<br />
backup-1:<br />
backup-3.nethere.net<br />
fpweb-1.nethere.net<br />
home-1.nethere.net<br />
koi.nethere.net<br />
marmaduke.inetworld.net<br />
ntdb-1.nethere.net<br />
ntweb-1.nethere.net<br />
ntweb-3.nethere.net<br />
ntweb-5.nethere.net<br />
ntweb-7.nethere.net<br />
phoenix.nethere.net<br />
unixweb-1.nethere.net<br />
unixweb-3.nethere.net<br />
unixweb-5.nethere.net<br />
unixweb-7.nethere.net<br />
shark.nethere.net<br />
tetra.nethere.net<br />
wms-1.nethere.net<br />
<br />
backup-2:<br />
andromeda.nethere.net<br />
backup-4.nethere.net<br />
eel.nethere.net<br />
fpweb-2.nethere.net<br />
ntdb-2.nethere.net<br />
ntweb-2.nethere.net<br />
ntweb-4.nethere.net<br />
ntweb-6.nethere.net<br />
unixweb-2.nethere.net<br />
unixweb-4.nethere.net<br />
unixweb-6.nethere.net<br />
ds.znet.com<br />
mx1.znet.com<br />
mx2.znet.com<br />
mx3.znet.com<br />
dmx.znet.com<br />
la.znet.com<br />
uf.znet.com<br />
<br />
backup-3:<br />
backup-1.nethere.net<br />
lists-1.nethere.net<br />
mailbox-1.mail.nethere.net<br />
mailbox-3.mail.nethere.net<br />
mta-1.mail.nethere.net<br />
mx-1.nethere.net<br />
nscache-1.nethere.net<br />
nsrbl-1.nethere.net<br />
pegasus.nethere.net<br />
scan-1.mail.nethere.net<br />
sndg-netapp-2.nethere.net<br />
relay-1.mail.nethere.net<br />
webmail-1.mail.nethere.net<br />
<br />
backup-4:<br />
ahi.nethere.net<br />
backup-2.nethere.net<br />
cp-1.nethere.net<br />
cygnus.nethere.net<br />
dragon.nethere.net<br />
lisa.nethere.net<br />
mailbox-2.mail.nethere.net<br />
mta-2.mail.nethere.net<br />
news-1.nethere.net<br />
nscache-2.nethere.net<br />
nsrbl-2.nethere.net<br />
sawfish.nethere.net<br />
scan-2.mail.nethere.net<br />
scribe.nethere.net<br />
sndg-netapp-1.nethere.net<br />
relay-2.mail.nethere.net<br />
urchin.nethere.net <br />
<br />
<br />
<br />
=== Mail servers ===<br />
==== Organization ====<br />
<pre><br />
<br />
<br />
NetHere Mail Servers<br />
====================<br />
<br />
<br />
<br />
<br />
Customers Inbound Mail Outbound Mail<br />
/\ || || /\<br />
|| || || smtp ||<br />
|| || \/ ||<br />
|| || ServerIron ||<br />
|| || || ||<br />
|| || || ||<br />
|| || smtp \/ ||<br />
|| || =========> mta-1 mta-2 ====> relay-1 <==== unixweb-* servers<br />
|| || || /\ relay-2 ntweb servers<br />
http || pop3 || || ||<br />
|| imap || \/ +--------------+<br />
|| || ServerIron |<br />
|| || || |<br />
|| || || |<br />
|| || \/ |<br />
|| || nsrbl-1 nsrbl-2 |<br />
|| || (spamhaus, dcc blacklist) |<br />
|| || || |<br />
|| \/ \/ |<br />
\/ mailbox-1 <=== scan-1 scan-2 |<br />
webmail <===> mailbox-2 (sendmail/amavisd/sophos) |<br />
imap mailbox-3 |<br />
/\ /\ |<br />
|| || \/<br />
nfs || +---------------------------> mta-db<br />
|| (LDAP)<br />
\/<br />
sndg-netapp-1<br />
sndg-netapp-2<br />
sndg-netapp-3<br />
<br />
<br />
</pre><br />
<br />
==== Overview ====<br />
mta-1.mail.nethere.net- Primary servers for all inbound and outbound mail.<br />
mta-1.mail.nethere.net- Primary servers for all inbound and outbound mail.<br />
<br />
nsrbl-1.nethere.net- Realtime blacklist check using dccd and spamhaus. The mta servers query the ip address of the mail server against the blacklist before transferring the email to the scanning servers.<br />
<br />
nsrbl-2.nethere.net<br />
<br />
scan-2.mail.nethere.net- Scans email with Sophos (currently disabled) and Spam Assasin before sending it to the mailbox servers.<br />
scan-2.mail.nethere.net Scans email with Sophos (currently disabled) and Spam Assasin before sending it to the mailbox servers.<br />
<br />
mailbox-1.mail.nethere.net- Front end server for the mail directories. Queries LDAP server for miscellaneous customer information including the directory the email should be stored in, passwords, spam sensitivity levels, etc. before delivering email to appropriate directory.<br />
mailbox-2.mail.nethere.net<br />
mailbox-3.mail.nethere.net<br />
<br />
Mail is stored in maildir format:<br />
<br />
/nfs/<nfs_number>/<platform>/u/s/username<br />
<br />
sndg-netapp-1-new.nethere.net- Solaris system using zfs set up as a nfs mount on mailbox-1, -2, and -3. Email is stored on these platforms for all customers.<br />
sndg-netapp-2-new.nethere.net<br />
sndg-netapp-3-new.nethere.net<br />
<br />
mx-1.nethere.net- spooling mail server<br />
<br />
mailx-1.nethere.net- all outbound mails originating from a web server is sent out this email server. All web servers are set to use mailx.nethere.net which resolves (through the serveriron) to either mailx-1.nethere.net and mailx-2.nethere.net. Since we shut down mailx-2, all outbound mail from the web servers goes through mailx-1. This was to prevent all the customers email from being blacklisted as a result of a web site being compromised.<br />
<br />
mta-db.mail.nethere.net- database server for the mta servers. All postfix databases are stored here.<br />
<br />
webmail.nethere.net- Hosts the webmail program for webmail.nethere.net, webmail.simplyweb.net and webmail.znet.net.<br />
<br />
==== Client mail settings ====<br />
<br />
Inbound Mail Server:<br />
<br />
POP3<br />
pop3.nethere.net<br />
pop3.znet.net<br />
pop3.simplyweb.net<br />
<br />
IMAP<br />
imap.nethere.net<br />
imap.znet.net<br />
imap.simplyweb.net<br />
<br />
Outbound Mail server:<br />
<br />
mail.nethere.net<br />
mail.znet.net<br />
mail.simplyweb.net<br />
<br />
Webmail:<br />
<br />
webmail.nethere.net<br />
webmail.znet.net<br />
webmail.simplyweb.net<br />
<br />
Control Panel:<br />
<br />
cp.nethere.net<br />
cp.znet.net<br />
cp.simplyweb.net<br />
<br />
==== nsrbl Disk Quota Warning ====<br />
<br />
In the event you get a nagios regarding space on the two nsrbl servers, <br />
it is most likely the result of the dccd databases getting too large. <br />
To clear out space, log into each nsrbl server and execute the following:<br />
<br />
df -h<br />
<br />
cd /var/dcc/libexec<br />
<br />
./stop-dccd<br />
<br />
cd /var/dcc<br />
<br />
rm dcc_db*<br />
<br />
sync ; sync<br />
<br />
cd /var/dcc/libexec<br />
<br />
./start-dccd<br />
<br />
ps auxw | grep dccd<br />
<br />
====Update outbound quota for a single ip address====<br />
<br />
We restrict the total number of emails that a customer can send to 2000 email recepients per 24 hour period. This is tracked by the ip address of the computer that connects to the mta server when it sends an outbound emal. The ip address and count is stored in the postfix database server, mta-db.mail.nethere.net. To increase the quota for a single ip address, you can log into either mta-1, mta-2, or console direcly into the database server. To use one of the mta servers:<br />
<br />
<pre><br />
1. Log onto mta-1.mail<br />
2. Switch to mysql<br />
$mysql -h 10.0.0.69 -u policyd_outbound -p<br />
username: policyd_outbound<br />
password: p0stf1x!<br />
3. Review database information<br />
mysql>show databases;<br />
4. Switch to the policyd table. <br />
mysql>use policyd;<br />
5. Review table header information<br />
mysql>show tables;<br />
mysql>describe throttle;<br />
6. Verify the ip address is above quota<br />
mysql>select * from throttle where _from='xxx.xxx.xxx.xxx'; #this is the ip address of the account you are increasing.<br />
7. Increase the maxium number of recepients the ip address can send to<br />
mysql>update throttle SET _rcpt_max = _rcpt_max + 20000 where _from='24.249.205.66';<br />
8. Verify<br />
mysql>select * from throttle where _from='xxx.xxx.xxx.xxx';<br />
</pre><br />
<br />
====Remove a blocked mail server ====<br />
<br />
The message in maillog helps identify where it is blocked.<br />
<br />
<pre><br />
Relaying denied due to excessive spam admin-2:/dist/files/nhmta/nethere/conf/postfix/common/client_reject<br />
Sender address rejected: Relaying denied due to Spam admin-2:/dist/files/nhmta/nethere/conf/postfix/common/sender_reject<br />
Relaying denied due to SPAM policyd???<br />
Client host rejected: ... listed at zen.spamhaus.org=127.0.0.2 nsrbl-1:/named/rbldns/zones/*.spamhaus.org<br />
status=sent (250 2.7.1 Ok, discarded, id=95886-33 - SPAM) user mail protection in provisioning????<br />
<br />
</pre><br />
<br />
====Remove a mail server from policyd blacklist====<br />
<br />
The ip address of a mail server can be placed on a blacklist for several reasons, to high of rate of inbound email from a single ip address, incorrectly formatted helo, etc. In the event you need to remove a mail server's ip from the policyd blacklist, you can log into either mta server of connect directly to the policyd database server, mta-db.mail.nethere.net. To remove the ip address using a mta server:<br />
<br />
<pre><br />
1. Log onto mta-1.mail<br />
2. Switch to mysql<br />
$mysql -h 10.0.0.69 -u policyd_inbound -p policyd<br />
username: policyd_inbound<br />
password: p0stf1x!<br />
3. Display database information<br />
mysql>show databases;<br />
4. Use policyd database<br />
mysql>use policyd;<br />
4. Display table information<br />
mysql>show tables;<br />
mysql>describe blacklist;<br />
5. Verify ip address has been blacklisted<br />
mysql>select * from blacklist where _blacklist='xxx.xxx.xxx.xxx'; <br />
6. Delete the entry<br />
mysql>delete from blacklist where _blacklist='xxx.xxx.xxx.xxx';<br />
7. Verify<br />
mysql> select * from blacklist where _blacklist='xxx.xxx.xxx.xxx';<br />
</pre><br />
<br />
The suspected mail server may also be in the helo table.<br />
<br />
<pre><br />
mysql> describe helo ;<br />
1. Check the IP address<br />
mysql> select * from helo where _host='xxx.xxx.xxx.xxx' ;<br />
2. Check the server helo name <br />
mysql> select * from helo where _helo like '%server%' ;<br />
3. Delete the entry<br />
mysql> delete from helo where _host='xxx.xxx.xxx.xxx' ;<br />
mysql> delete from helo where _helo like '%server%' ;<br />
</pre><br />
<br />
====Remove a mail server from spamhaus blacklist====<br />
<br />
ssh to nsrbl-1.mail and nsrbl-2.mail<br />
<br />
<pre><br />
<br />
cd /named/rbldns/zones <br />
<br />
find which file the IP Address is in.<br />
<br />
grep <IP Addr> *.spamhaus.org<br />
co -l <xxx.spamhaus.org><br />
vi xxx.spamhaus.org<br />
ci -u <xxx.spamhaus.org><br />
<br />
</pre><br />
<br />
====Add ip address to client_reject====<br />
<br />
The client_reject list is one of the ways we combat spam. If we receive spam compaints from a mail server or otherwise identitfy a mail server as a source of spam, we can add it to a client_reject list. An email sent from a mail server on the client_reject list is rejected with a notice saying "Relaying denied due to excessive spam". To add an ip address to the list, we use admin-2. Once the list is updated on admin-2, it is pushed out to mta-1 and mta-2. To add an ip address on admin-2:<br />
<br />
<pre><br />
Log into admin-2.nethere.net.<br />
bash-3.00# cd /dist/files/nhmta/nethere/conf/postfix/common<br />
bash-3.00# co -l client_reject<br />
RCS/client_reject,v --> client_reject<br />
bash-3.00# vi client_reject<br />
<br />
-Add the ip address to the bottom of the list in the appropriate format. It you want to reject any email from the mail server, you would use:<br />
xxx.xxx.xxx REJECT Relaying denied due to excessive spam<br />
where xxx.xxx.xxx is the /24 subnet of the mail server the spam originated from. Do not place a period after the last octet. For example, to add a /16 subnet you would use:<br />
xxx.xxx REJECT Relaying denied due to excessive spam<br />
The REJECT key word tells postfix to reject the email. If you want to allow email from a /24 subnet, use the following format:<br />
xxx.xxx.xxx OK<br />
This tells postfix to accept email from that subnet.<br />
<br />
bash-3.00# ci -u client_reject<br />
RCS/client_reject,v <-- client_reject<br />
new revision: 1.286; previous revision: 1.285<br />
enter log message, terminated with single '.' or end of file:<br />
>> .<br />
done<br />
bash-3.00# make<br />
/nethere/software/nhmta/postfix-2.4.5/sbin/postmap -C . client_reject<br />
mv client_reject.db maps/hash/client_reject.db<br />
bash-3.00# cd /dist/rdist<br />
bash-3.00# gmake nhmta-update<br />
updating host mta-1.mail.nethere.net<br />
updating: /dist/files/nhmta//nethere/conf/postfix/common/maps/hash/client_reject.db<br />
updating: /dist/files/nhmta//nethere/conf/postfix/common/client_reject<br />
updating host mta-2.mail.nethere.net<br />
updating: /dist/files/nhmta//nethere/conf/postfix/common/maps/hash/client_reject.db<br />
updating: /dist/files/nhmta//nethere/conf/postfix/common/client_reject<br />
bash-3.00#<br />
</pre><br />
<br />
====Free mysql disk space on mta-db.mail.nethere.net====<br />
<br />
In response to nagios alert "[Nagios] PROBLEM alert - mta-db.mail.nethere.net/Disk mysql is WARNING", to clear space within the mysql database:<br />
<br />
1. ssh to mta-db.mail.nethere.net.<br />
<br />
2. Log into mysql and execute the following:<br />
<br />
[root@mta-db user]# mysql -uroot -p<br />
<br />
Enter password: dB@dm1N!<br />
<br />
mysql> show databases;<br />
<br />
mysql> use policyd;<br />
<br />
mysql> show tables;<br />
<br />
mysql> describe helo;<br />
<br />
mysql> describe throttle;<br />
<br />
mysql> describe throttle_from_instance;<br />
<br />
mysql> select * from throttle_from_instance limit 10;<br />
<br />
mysql> select now();<br />
<br />
mysql> select unix_timestamp(now());<br />
<br />
mysql> select count(*) from throttle_from_instance where _expire > 1426534697;<br />
<br />
mysql> select max(_expire) from throttle_from_instance;<br />
<br />
mysql> truncate table throttle_from_instance;<br />
<br />
mysql> select count(*) from throttle_from_instance;<br />
<br />
mysql> select * from throttle_from_instance;<br />
<br />
mysql> show tables;<br />
<br />
mysql> optimize table throttle_from_instance;<br />
<br />
mysql> show databases;<br />
<br />
mysql> use policyd;<br />
<br />
mysql> quit<br />
<br />
[root@mta-db user]# df -h<br />
<br />
== Errors ==<br />
<br />
<br />
=== Common webmail errors ===<br />
<br />
<br />
Error:<br />
--cut--<br />
Fatal error: Call to undefined function: applicatio€”¬p() in<br />
/www/webmail.nethere.net/htdocs/x/m/templates/message/navbar.inc on line 7<br />
--cut--<br />
<br />
Solution: Restart apache on webmail-1 (apachectl restart)<br />
<br />
<br />
Error:<br />
--cut--<br />
ERROR There was an error sending your message: unable to add recipient<br />
[webhosting@nethere.com]: Invalid response code received from server<br />
--cut--<br />
<br />
Solution: More than likely DNS/domain issue, but check mta-1.nethere.net to be sure:<br />
--cut--<br />
Aug 30 15:08:54 mta-1 nh/smtpd[50501]: NOQUEUE: reject: RCPT from webmail-1.mail.nethere.net[66.63.128.181]: 450 <orders@mulligrins.com>: Sender address rejected: Domain not found; from=<orders@mulligrins.com> to=<webhosting@nethere.com> proto=ESMTP helo=<webmail.nethere.net><br />
--cut--<br />
Problem: Webmail shows "1-5" messages but inbox appears empty. POP shows messages #1,2,3,4,5... could not be retrieved... server response error cannot open the message file it's gone.<br />
<br />
Solution: Ownership issue<br />
<br />
1) Use the provisioning tool to determine mailbox location:<br />
a) Click on email account, Engineering<br />
b) Location is the "Storage Path:" <br />
<br />
2) Log in to either mail server then change ownership of maildir to mailuser:mailuser<br />
# chown -R mailuser:mailuser /nfs/#/platform/u/s/username<br />
<br />
<br />
<br />
== DEPRECIATED INFORMATION ==<br />
<br />
<br />
<br />
=== Webservers ===<br />
<br />
<br />
<br />
==== Allowing SSI (Server Side Includes) ====<br />
<br />
***** NOTE: This has been DEPRECIATED, all servers now allow SSI by default *****<br />
<br />
We do not allow EXEC permissions for SSI.<br />
<br />
1) Check out Apache config for www.domain.tld in /nethere/conf/apache/conf/<vhosts,nvhosts><br />
# co -l www.domain.tld(.common) <br />
<br />
2) Add the following lines under the <Directory> directive<br />
--cut--<br />
AddType text/html .shtml<br />
AddHandler server-parsed .shtml<br />
--cut--<br />
<br />
3) Edit the "Options" line to be the following:<br />
--cut--<br />
Options Indexes FollowSymLinks IncludesNOEXEC<br />
--cut--<br />
<br />
4) Add a DirectoryIndex line above the <Directory> directive with the following:<br />
--cut--<br />
DirectoryIndex index.shtml index.html index.htm home.html home.shtml index.php<br />
--cut--<br />
<br />
** In summary, the config should look something similar to this **<br />
--cut--<br />
DocumentRoot "/www/www.domain.tld/htdocs"<br />
DirectoryIndex index.shtml index.html index.htm home.html home.shtml <br />
<Directory "/www/www.domain.tld/htdocs"><br />
AddType text/html .shtml<br />
AddHandler server-parsed .shtml<br />
Options Indexes FollowSymLinks IncludesNOEXEC<br />
Order allow,deny<br />
Allow from all<br />
</Directory><br />
--cut--<br />
<br />
5) Check in Apache config<br />
# ci -u www.domain.tld(.common) < /dev/null <br />
<br />
6) Restart Apache<br />
# apachectl configtest<br />
# apachectl restart<br />
<br />
More SSI notes can be found here:<br />
http://httpd.apache.org/docs/1.3/howto/ssi.html<br />
<br />
= SimplyInternet =<br />
<br />
= Z-net =<br />
<br />
= Migrating to LAMP Host Servers =<br />
<br />
Below are outlines of the steps to migrate website & email hosting from nethere to lamphost configured servers. This is almost certainly incomplete at this stage. The exact steps to follow can vary depending on specifics of the client, so use this as a guide only. Be ready to adjust commands or add steps as needed.<br />
<br />
== Edit Nethere DNS ==<br />
<br />
ssh user@admin-1.nethere.net<br />
<br />
sudo -i<br />
<br />
ssh ns1.nethere.net<br />
<br />
cd /named/master<br />
<br />
co -l example.com<br />
<br />
vi example.com<br />
<br />
ci -u example.com<br />
<br />
rndc reload example.com<br />
<br />
Make sure to increment serial number by 1 each time a change is made.<br />
<br />
== Migrate Website to nh3.jcihosting.net ==<br />
<br />
Set TTL to 300 on ns1.nethere.net<br />
<br />
<br />
Open up these web admins to get current account/site info:<br />
<br />
http://toolbox.nethere.net/ -> Provisioning Tool (search for domain name)<br />
<br />
https://secure.johncompanies.com/mgmt/index.html (search for col#####)<br />
<br />
<br />
1. Create the lamphost user.<br />
<br />
https://www.lamphost.com/admin/user/user/create<br />
<br />
username: col#####<br />
<br />
email: same as listed for col##### account<br />
<br />
password: random, secure, won't be shared with client<br />
<br />
defaults on rest of page<br />
<br />
<br />
2. Create the customer account<br />
<br />
https://nh1.jcihosting.net:8443/<br />
<br />
https://nh2.jcihosting.net:8443/<br />
<br />
https://nh3.jcihosting.net:8443/<br />
<br />
https://nh4.jcihosting.net:8443/<br />
<br />
LAMP Host Users -> Add User -> col#####<br />
<br />
<br />
Virtual Hosts -> Add Virtual Host<br />
<br />
Username: Select same username as above from pulldown<br />
<br />
Domain: example.com<br />
<br />
Hosting Plan: select plan that matches what's listed in https://secure.johncompanies.com/mgmt/view.html?cid=col#####<br />
<br />
Add default DNS: No<br />
<br />
<br />
3. Virtual Hosts -> Site Manager (next to domain added)<br />
<br />
Site Manager Users -> Add User<br />
<br />
Username: can be anything - I typically use Optigold Login from nethere admin<br />
<br />
Password: make up something secure and record - this one will go to the user<br />
<br />
<br />
4. Email & FTP -> Add Account<br />
<br />
Email Account: ftp<br />
<br />
Password: make up something secure and record - this one will go to the user<br />
<br />
Enable FTP Access: Yes<br />
<br />
<br />
5. Optional: Add MySQL database if site needs it (WordPress etc)<br />
<br />
<br />
ssh to nh3.jcihosting.net and run 'add_mysql.php <example.com>'<br />
<br />
MySQL database info will be emailed to webmaster@lamphost.com (and output on command line - ignore instructions to run additional commands)<br />
<br />
https://mail.lamphost.com<br />
Sm77DdnQ<br />
<br />
6. Copy over website<br />
<br />
ssh nh3<br />
<br />
cd /var/www/example.com/<br />
<br />
rsync -v --archive --one-file-system --delete --delete-during --rsh=/usr/bin/ssh dsmith@unixweb-10.nethere.net:/www/www.example.com/htdocs ./<br />
<br />
Change 'matt' to your username. Note that I had to add my username to groups httpd & webuser to allow me to copy website files without being root.<br />
<br />
chown -R col#####:col##### htdocs<br />
<br />
<br />
7. If site has MySQL data find the connection info. For example, WP sites will have the info in htdocs/wp-config.php<br />
<br />
mysqldump -h mysqldb-1.webhost.nethere.net -u username --password=password databasename > databse.sql<br />
<br />
mysql exampledotcom < database.sql<br />
<br />
rm database.sql<br />
<br />
<br />
8. If site is ready to be made live on nh3.lamphost.net then update DNS on ns1.nethere.com<br />
<br />
Send email to client with details. Search support@jcihosting.com Sent folder for emails with Subject "updated hosting for" for example emails. You will have to adjust as needed (username/passwords, different info based on specific client).<br />
<br />
9. After you finish, update the johncompanies.com backend for the col0# CID and change the nethere hosting server to nh3.jcihosting.net.<br />
<br />
10. To add a SSL certificate<br />
letsencrypt-create.php <domain.com><br />
<br />
To remove a SSL certificate<br />
del-letsencrypt-certificate.php <domain.com><br />
<br />
== Migrate Email to nh3.jcihosting.net ==<br />
<br />
If migrating website & emails both follow the above steps to transfer the website first.<br />
<br />
If migrating emails only the follow the first steps 1-3 of the above steps to set up the virtualhost on nh3.<br />
<br />
<br />
1. Set TTL for MX record to 300.<br />
<br />
<br />
2. Generate email report:<br />
<br />
Log in to http://toolbox.nethere.net/<br />
<br />
Select Provisioning Tool and search for the domain<br />
<br />
Navigate to the Email section and then Generate Report.<br />
<br />
Select these fields for the report:<br />
<br />
Mail address<br />
<br />
Alias<br />
<br />
Forwarding address<br />
<br />
Mail storage path<br />
<br />
Copy the result of the report and save it as a text file in /tmp/ on nh3.jcihosting.net named example.com.txt (replace domain name)<br />
<br />
<br />
3. Create the e-mail accounts<br />
<br />
Make sure the virtualhost has enough email accounts allocated before proceeding. <br />
Edit the virtualhosts.virtualhost_settings table directly if need be to increase allocation.<br />
<br />
Log in here: https://nh3.jcihosting.net:8443/<br />
Select "MySQL.LH"<br />
Go to the "virtualhosts" database<br />
Search the "virtualhost_settings" settings for 'example.com' <br />
and adjust the 'email_accounts' setting.<br />
<br />
<br />
Run this script once and only once:<br />
<pre><br />
/usr/local/bin/migrate_mail_accounts_from_nethere.php example.com<br />
</pre><br />
<br />
That will create all email accounts and aliases.<br />
<br />
A file containing the list of email accounts and their new passwords will <br />
be created as /tmp/example.com-passwords.txt. Grab a copy of that for <br />
sending to the client.<br />
<br />
<br />
4. Sync the emails<br />
<br />
Another file will be created that contains the bash commands to sync emails from the old server. This will be named /tmp/example.com-sync.sh<br />
<br />
Copy the file somewhere like the /root/ home dir and add "#!/bin/bash" as the first line and change perms to 750. Now you can run this script to sync all mails from nethere to nh3. Run it immediately to get an initial sync of emails.<br />
<br />
When ready to switch email hosting over run the sync script one more time, then update DNS on nh3.nethere.net.<br />
<br />
If you'd like to sync again after the move (if there was a delay or you think there may have been incoming emails during the transtion) remove the '--del' option from the sync script and then re-run.<br />
<br />
<br />
5. Send email to customer<br />
<br />
Search support@jcihosting.com Sent folder for messages with Subject "new email hosting for" for examples of emails to use for communicating with the client.<br />
<br />
6. Update DNS and put NetHere mail on hold<br />
Once DNS has been pointed to the new server, go into the NetHere Toolbox http://toolbox.nethere.net/ and go to Domain -> General and then set Status to 'On Hold'.<br />
<br />
7. Update host in JohnCompanies database.<br />
<br />
== Moving between Host Servers on Lamphost ==<br />
<br />
You should delete the old virtual host before creating the new<br />
virtual host. It may take up to an hour to delete the old virtual<br />
host. To speed up the delete:<br />
<pre><br />
/opt/lamphost/cron-bin/del_virtualhosts.php<br />
</pre><br />
<br />
If you create the new virtual host before, you will need to update<br />
the mylampsite.com:<br />
<pre><br />
/opt/lamphost/bin/update_mylampsite_subdomain.php databaseproviders.com<br />
</pre><br />
<br />
== Upgrade a Lamphost server to increase database limit ==<br />
<br />
I've increased the database limit for this customer to 2. Now they can add a second db via the Site Manager admin.<br />
<br />
<pre><br />
Here is how I made the adjustment (we don't have a friendly admin for this function):<br />
Log in to Server Manager: https://nh3.jcihosting.net:8443/<br />
Select the "MySQL.LH" phpMyAdmin<br />
Select 'virtualhosts' db.<br />
Adjust the 'databases' setting for the argee.com domain in the 'virtualhost_settings' table.<br />
</pre><br />
<br />
== looking in the mail log files ==<br />
<br />
exigrep</div>75.80.128.147https://wiki.jcihosting.com/index.php?title=VPS_Management&diff=2238VPS Management2020-07-02T19:52:11Z<p>75.80.128.147: /* Ganeti Install a FreeBSD VPS */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
== Moving a server to a different account in the management system ==<br />
<br />
This is a tough one. You gotta touch a bunch of tables in the mysql database.<br />
<pre><br />
systems<br />
subs<br />
ipmap<br />
</pre><br />
<br />
basically, get the sysid and go off that:<br />
<br />
<pre><br />
update systems set cid='newcid' where sysid=xxxxx<br />
update subs set cid='newcid' where sysid=xxxxx<br />
update ipmap set cid='newcid' where sysid=xxxxx<br />
</pre><br />
<br />
You then need to rename the server disk space to change the old ID to the new one.<br />
<br />
For a jail -<br />
<pre><br />
stopjail <hostname><br />
mv 69.55.2XX.XXX-col0<oldid> 69.55.2XX.XXX-col0<newid><br />
mv 69.55.2XX.XXX-col0<oldid>-DIR 69.55.2XX.XXX-col0<newid>-DIR<br />
vi /usr/local/jail/bin/quad1 [Update the ID in the quad file]<br />
startjail <hostname><br />
</pre><br />
<br />
For a virt -<br />
You will have to <br />
* stop the virt <br />
* rename the directories <br />
* modify the config file /vzconf/<veid>.conf <br />
* rename the virt conf file <br />
* restart the new virt.<br />
<br />
that ought to do it..<br />
<br />
<br />
== Re-enable a customer account in the management system ==<br />
<br />
For example to re-instate col03438, In mysql:<br />
<br />
<pre><br />
show tables ;<br />
select * from customers where cid='col03438' ;<br />
update customers set status=1 where cid='col03438' ;<br />
select * from systems where cid='col03438' ;<br />
update systems set stop_date=NULL where cid='col03438' ;<br />
update systems set cancel_date=NULL where cid='col03438' ;<br />
update systems set status=1 where cid='col03438' ;<br />
select * from subs where cid='col03438' ;<br />
update subs set stop_date=NULL where cid='col03438' ;<br />
update subs set status=1 where cid='col03438' <br />
<br />
</pre><br />
<br />
= Ganeti VPS =<br />
<br />
This is how to create a VPS on the Lamphost ganeti cluster.<br />
I have created scripts for creating a FreeBSD VPS and one for<br />
creating a Linux VPS.<br />
<br />
== Ganeti Install a Linux VPS ==<br />
<br />
To create a linux VPS use<br />
<pre><br />
/root/new <name> <disk size> <memory> <IP Addr> [<OS>]<br />
<br />
( for example /root/new vps.example.com 30 2 69.55.238.10 centos-8 )<br />
</pre><br />
<br />
You need to customize the install by adding a "jci" account and a customer<br />
account (col0XXXX) as well. Make sure the customer acct (col0XXXX) is<br />
in a group to allow sudo.<br />
<br />
== Ganeti Install a FreeBSD VPS ==<br />
<br />
To create a FreeBSD VPS, we have to do a full install. This will <br />
create a new VPS. on gn1, do<br />
<br />
<pre><br />
/root/newfreebsd12 <name> <disk size> <memory> <IP Addr> [<OS>]<br />
</pre><br />
<br />
You will then have to do the install via VNC.<br />
<br />
<pre><br />
gnt-instance info <name> | grep -i connection<br />
</pre><br />
<br />
Use your favorite VNC client (I use Tiger VNC). to connect <br />
<br />
<br><br />
You need to customize the install by adding a "jci" account and a customer <br />
account (col0XXXX) as well. <br />
<br><br />
Make sure the customer acct (col0XXXX) is in a <br />
group to allow sudo (wheel). <br />
<br />
<br> <br />
You will need to install sudo.<br />
<br />
<pre> <br />
pkg install sudo # install sudo package<br />
visudo # to add wheel account to sudoers<br />
vi /etc/group # add jci and user acct to wheel<br />
</pre><br />
<br />
<br><br />
You will also have to set up a console in /etc/ttys.<br />
<pre><br />
vi /etc/ttys<br />
(change onifconsole to on for ttyu0).<br />
</pre><br />
<br />
== Ganeti restart a VPS ==<br />
<br />
To shutdown a ganeti VPS<br />
<br />
<pre><br />
gnt-instance reboot <VPSname><br />
</pre><br />
<br />
<br />
== Ganeti shutdown a VPS ==<br />
<br />
To shutdown a ganeti VPS<br />
<br />
<pre><br />
gnt-instance shutdown <VPSname><br />
</pre><br />
<br />
To permanently delete a ganeti VPS<br />
<br />
<pre><br />
gnt-instance remove <VPSname><br />
</pre><br />
<br />
== Ganeti Re-Intsall a VPS ==<br />
<br />
Re-installing the OS will destroy all data on the VPS.<br />
Login to the Ganeti master node (currently gn1).<br />
<br />
To reinstall the OS on a VPS the VPS must be shutdown.<br />
<pre><br />
gnt-instance shutdown <VPSname><br />
</pre><br />
<br />
Do the reinstall.<br />
<pre><br />
gnt-instance reinstall <VPSname><br />
</pre><br />
<br />
Restart the VPS.<br />
<pre><br />
gnt-instance startup <VPSname><br />
</pre><br />
<br />
Now you need to do the customization for the VPS. This is<br />
basically the same as the Dedicated server customization.<br />
<br />
https://69.55.229.8/index.php/New_Signups#Dedicated_Server_Install_Checklist<br />
<br />
== Ganeti login to a VPS ==<br />
<br />
To login to a ganeti VPS<br />
<br />
<pre><br />
gnt-instance console <VPSname><br />
<carriage return><br />
<carriage return><br />
<carriage return><br />
</pre><br />
<br />
Use jci and appropriate Ganeti jci password.<br />
<br />
or you can ssh to the VPS name or IP address.<br />
<br />
= FreeBSD 10.1 VPS (jail5 and jail6) =<br />
<br />
This is information on how to manage the FreeBSD 10.1 VPS servers on Jail5.<br />
<br />
<br />
== Making new customer VPS ==<br />
<pre><br />
virtmake IP[,IP] <CID> hostname ipfw# email {FSP|FB|FS|FM|FP} [size in GB]<br />
</pre><br />
<br />
Note: <br />
For now, CID must be unique in the last 2 digits of the CID.<br />
<br />
Must do a "vm start <CID>" after creating a VPS.<br />
<br />
== Starting VPS ==<br />
To start all vms on the system<br />
<pre><br />
vm start<br />
</pre><br />
<br />
To start a single VPS <br />
<pre><br />
vm start <CID><br />
</pre><br />
<br />
== List running VPS ==<br />
<pre><br />
vm status<br />
</pre><br />
<br />
== Attach to a user VPS console ==<br />
<pre><br />
vm attach <CID><br />
</pre><br />
<br />
<br />
To detach from a console, type tilde control-d (~ ctl-d).<br />
<br />
== Stop a VPS ==<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
== Restart a VPS ==<br />
<pre><br />
vm restart <CID><br />
</pre><br />
<br />
<br />
== Modify a VPS ==<br />
<br />
To modify a VPS config, stop the VPS<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
Mount the VPS data file<br />
<pre><br />
vm mount <CID><br />
<br />
cd /data/vm/private/<CID>/mount<br />
</pre><br />
<br />
Modify the user files necessary.<br />
<br />
Restart the VPS.<br />
<br />
<pre><br />
vm umount <CID><br />
<br />
vm start <CID><br />
</pre><br />
<br />
== Notes on a new FBSD 10.x VPS (jail6) ==<br />
<br />
<pre><br />
#! /bin/sh<br />
#<br />
# new <colo#> <disk><br />
<br />
<br />
memory=1024<br />
<br />
x="$1"<br />
<br />
disk="$2"<br />
if [ "y" == "y$disk" ] ; then<br />
disk=60<br />
fi<br />
<br />
disk2=`expr $disk - 4`<br />
<br />
<br />
vm image provision 04dcdf96-d843-11e6-b131-001e67fcc64f col0$x<br />
# vm configure col0$x<br />
sed -e "s/memory=256M/memory=${memory}M/" /vms/col01114/col01114.conf >/vms/col0$x/col0$x.conf<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
# vm start col0$x<br />
<br />
<br />
exit<br />
<br />
vm console col0$x<br />
<br />
#<br />
# to increase the customer disk<br />
#<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
<br />
## on the customer VPS<br />
<br />
gpart recover vtbd0<br />
gpart show vtbd0<br />
swapoff /dev/vtbd0p3<br />
gpart delete -i 3 vtbd0<br />
gpart show vtbd0<br />
sysctl kern.geom.debugflags=16<br />
gpart resize -i 2 -a 4k -s $disk2\G vtbd0<br />
<br />
growfs /dev/vtbd0p2<br />
<br />
gpart show vtbd0<br />
gpart add -t freebsd-swap -a 4k vtbd0<br />
gpart show vtbd0<br />
swapon /dev/vtbd0p3<br />
<br />
df -h<br />
reboot<br />
<br />
vi /etc/rc.conf<br />
hostname<br />
IP Addr<br />
passwd root<br />
passwd user<br />
clear history<br />
cp /dev/null /var/log/utx.log<br />
cp /dev/null ~root/.history<br />
history -c<br />
<br />
</pre><br />
<br />
= FreeBSD Jails =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Making new customer jail 4.x ==<br />
<br />
1. run [[#js|js]] to figure out which partition, IP to put it on, find an unused vn, and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vnX hostname shorthost quad/safe# ipfw# email [gb disk, default 4]</tt><br />
<br />
jail14# /tmp/jailmake 69.55.226.152 /mnt/data2/69.55.226.152-col00182 vn23 negev.cerastes.org negev 4 182 cerastes@cerastes.org<br />
<br />
<br />
== Making new customer jail 6.x ==<br />
<br />
1. run [[#js|js]] to figure out which gvinum vols are free (and which mnt (disk) each gvinum goes with/to), IP to put it on and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vN[,vN] hostname shorthost quad/safe# ipfw# email</tt><br />
<br />
jail19# jailmake 69.55.236.17 /mnt/data1/69.55.236.17-col01574 v10 uncle-leo.commadev.com uncle-leo 1 1574 lee@commadev.com,lee@gmail.com<br />
<br />
<br />
== Changing an ip for freebsd VPS ==<br />
<br />
*stopjail (hostname)<br />
*on systems using vnfile or mdfile: rename vnfile/mdfile, using new ip<br />
*rename dir using new ip<br />
*Edit quad (make sure to update all lines: directory, vn/mdfile)<br />
*buildsafe (not necessary on systems having only quad1)<br />
*if you're adding an ip not already on the host machine: <tt>ipadd x.x.x.x</tt><br />
*startjail (hostname)<br />
if backups: <tt>mvbackups</tt> ...<br />
*Edit dir/ip in mgmt<br />
*if any rules, update any firewall rules to use new ip (careful not to make sure there arent multiple rules per ipfw#- search by colo#)<br />
*(if customer asks and has domains) update their domains on ns1c<br />
*optional: update ipfw counters on the host server<br />
<br />
<br />
== Rename a gconcat vol ==<br />
<br />
*stop jail, unmount everything<br />
*gconcat stop vol<br />
*gconcat clear /dev/gvinum/vN<br />
*gconcat clear /dev/gvinum/vN+1<br />
*gconcat label -v newname /dev/gvinum/vN /dev/gvinum/vN+1<br />
*bsdlabel /dev/concat/newname<br />
(make sure a partition is not fstype unused, if so change to: 4.2BSD)<br />
<br />
<br />
== Remaking a VPS (on same jail) ==<br />
<br />
# take him down (stopjail)<br />
# make note of host, vn/gvin/md, ip<br />
# remove from quad/safe<br />
# move vnfile (rename)<br />
# remove dir (jailmake will make a new one)<br />
# remove ipfw counts<br />
# jailmake<br />
# update backup script<br />
# update db with new dir/ip (use the “move” link, mark old as stopped today)<br />
# update firewall if changed ip<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
<br />
== Adding disk to a 4.x jail ==<br />
<br />
NOTE: if you unmount before vnconfig, re-vnconfig then unmount then vnunconfig<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them.<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to dump to a different partition and create the new vn on the current partition. If no space to make make dump, then move old vn’s to backup or worse, dump to backup.<br />
<br />
<br />
2. <tt>g <customerID></tt> to get the info<br />
<br />
<br />
3. make vnfile, newfs and mount. USE CAUTION not to overwrite existing vn. <br><br />
Ex: <tt>vnconfig –T -S 9g -s labels -c /dev/vn22 /mnt/data2/65.214.160.117-col00496<br><br />
disklabel -r -w vn30 auto<br><br />
newfs /dev/vn30c<br><br />
mkdir /mnt/data2/65.214.160.117-col00496-DIR<br><br />
mount /dev/vn30c /mnt/data2/69.55.237.26-col00241-DIR<br><br />
cd /mnt/data2/65.214.160.117-col00496-DIR</tt><br />
<br />
<br />
4. <tt>jailkill <hostname></tt><br />
<br />
<br />
5. <tt>dump -0a -f - /dev/vn1 | restore -r -f -</tt> <br />
<br />
<br />
6. <tt>rm restoresymtable</tt><br />
<br />
<br />
7. unmount and unvnconfig old system:<br><br />
<tt>umount /dev/vnNNc <br><br />
vnconfig –u /dev/vnNN</tt><br />
<br />
<br />
8. move vnfile. Ex: <tt>mv /mnt/data1/69.55.237.26-col00241 /mnt/data1/old-col00241-vnfile-20110312-noarchive</tt><br />
<br />
<br />
9. remove old dir: <tt>rmdir /mnt/data1/69.55.237.26-col00241-DIR</tt><br />
<br />
<br />
10. edit quad/safe to point to new location, run <tt>[[#buildsafe|buildsafe]]</tt><br />
<br />
<br />
11. start jail: <tt>[[#startjail|startjail]] <hostname></tt><br />
<br />
<br />
12. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
13. mv backups if necessary<br />
<br />
WARNING: if you restore to unmounted vn then you’re actually restoring to /mnt/data1<br />
you can tar and do above to recover<br />
<br />
NOTE: you can’t move a customer to a system that has a base sys that is diff from the one they came from (BSD 4.8 to 4.5 = won’t work)<br />
<br />
NOTE: we don’t mount procfs's to anyone’s system by default.<br />
<br />
NOTE: mount_nulls: <br />
also goofy mount_nulls seen in df -k on jail2<br><br />
also needs to be done in safe and quad<br><br />
if he wants another, use safe/quad as example and add it to file. also exec command at prompt to add mount (no reboot, remount or rejail necessary)<br />
<br />
== Moving customer to another jail machine ==<br />
<br />
Systems should only be moved between similarly-versioned jails. If you cannot get a match, you should go to freebsd.org and see what the differences were between the version you're moving from and the new version. Many of the updates will be to drivers and kernels - these don't affect the customer. Perhaps they updated traceroute or a library...you should copy the new/changed files into the VPS/jail on the new host. An alternate method for moving to a different version would be to buildworld to bring their full distribution up to date. Optional steps for this kind of upgrade are included below.<br />
<br />
1. <tt>g <customerID></tt><br />
<br />
<br />
2. <tt>[[#jailkill|jailkill]] <hostname></tt><br />
<br />
<br />
3. create new device on target system<br><br />
4.x:<br><br />
<pre>vnconfig –T –S 4g –s labels –c /dev/vn1 /mnt/data1/69.55.22x.x-col00XXX<br />
disklabel –r –w vn1 auto<br />
newfs /dev/vn1c</pre><br />
<br />
6.x:<br><br />
<pre>bsdlabel -r -w /dev/gvinum/v1<br />
newfs /dev/gvinum/v1a<br />
or <br />
gconcat label v1-v3 /dev/gvinum/v1 /dev/gvinum/v2 /dev/gvinum/v3<br />
bsdlabel -r -w /dev/concat/v1-v3<br />
newfs /dev/concat/v1-v3a</pre><br />
<br />
7.x+:<br><br />
Run jailmakeempty then skip step 4 & 7 below<br />
<br />
<br />
4. make and mount the dir on new system. <br><br />
Ex: <pre>mkdir /mnt/data2/69.55.230.3-col00123-DIR<br />
chmod 755 /mnt/data2/69.55.230.3-col00123-DIR<br />
mount [device] /mnt/data2/69.55.230.3-col00123-DIR</pre><br />
<br />
<br />
5. [[#stopjail|stopjail]] <hostname> 1<br />
<br />
<br />
6. dump fs to new system. ex: <tt>[[#dumpremoterestore|dumpremoterestore]] /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR</tt> (make sure you can ssh as root on remote machine) <br />
<br />
<br />
6a. OPTIONAL BUILDWORLD:<br><br />
cd /usr/src<br />
make world DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
cd etc<br />
make distribution DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
(you may have to rm an openssh file and re-make dist)<br />
<br />
rm -rf /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/periodic/daily/400.status-disks<br />
vi /etc/periodic/security/100.chksetuid<br />
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt><br><br />
with: <tt>MP='/' (use single quotes)</tt><br />
<br />
Ask user if they want ports overwritten with current, if yes:<br />
cp -r /usr/ports /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/usr<br />
<br />
on source: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d .<br />
(copy in all info)<br />
<br />
on target: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d . <br />
(paste all info)<br />
<br />
on source: <br />
cat /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group<br />
(copy in all info)<br />
<br />
on target:<br />
cat > /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group <br />
(paste all info)<br />
<br />
<br />
7. edit quad on source system copy over entries for the jail to the target system <br />
take care that the vn/gvinum/gconcat/md devices on the target system are’t in use and that the /mnt/dataN path doesn’t need changing<br />
<br />
<br />
8. run [[#buildsafe|buildsafe]] (if this is <=6.x) on target system (copies changes made to quad into safe file)<br />
<br />
<br />
9. remove ip from source system. Ex: <tt>ipdel 69.55.230.3</tt><br />
<br />
<br />
10. add ip to target system. Ex: <tt>ipadd 69.55.230.3</tt><br />
<br />
<br />
11. start new system, with [[#startjail|startjail]] or manually, pasting entries found from running <tt>g <customerID></tt> on the new system<br />
<br />
<br />
12. run <tt>[[#canceljail|canceljail]] col0xxxx</tt> on source system. This should prompt you to remove backups if any existed. Do not let it do this, rather:<br />
<br />
<br />
13. if backups existed move them to the new host via the <tt>[[#mvbackups_.28freebsd.29|mvbackups]] script</tt><br />
<br />
<br />
14. edit quad on source system, edit comment to reflect a move rather than a cancel, ex: <tt># moved to jail2 col00241</tt><br />
<br />
<br />
15. edit mgmt to reflect new host and dir for new system<br />
<br />
<br />
16. optional add ipfw rules on new system. Ex:<br />
<tt>ipfw add 01231 count ip from 69.55.230.2 to any<br />
ipfw add 01232 count ip from any to 69.55.230.2</tt><br />
<br />
== Increasing inodes for a VPS ==<br />
<br />
when doing the newfs: <br />
newfs -i 4096 ...<br />
<br />
<br />
== Jail services unable to contact services within same jail ==<br />
<br />
This is due to a messed up routing table (only seen it on jail2 before). <br />
<br />
route delete 69.55.228.65/32 route add -net 69.55.228.65/32 -iface fxp0 -nostatic -cloning route add 69.55.228.65 -iface lo0<br />
<br />
<br />
== /dev/null permission resets == <br />
<br />
Applies to 4.x jails only<br />
<br />
For some reason that I do not understand at all, the /dev/null node in customer jails, after the system is restarted, often reverts permissions to 0600 ... which is bad, because non-root processes cannot redirect to /dev/null anymore. In fact, a fair number of server daemons will complain and/or fail if this is the case.<br />
<br />
So, after a restart, after all systems have restarted, run postboot, which does something like:<br />
<br />
for f in `df -k | grep /dev/vn | awk '{print $6}'` ; do chmod 0666 $f/dev/null ; done<br />
<br />
Later, after the system is up, if anyone ever complains about their /dev/null permissions for any reason ... really if anyone mentions<br />
/dev/null in any way, tell them to run:<br />
<br />
chmod 0666 /dev/null<br />
<br />
We have ultimately fixed this by adding chmods to the quad/safe's<br />
<br />
<br />
== mknod in a jail == <br />
<br />
For a customer who wants to run a chroot name server (named) in a jail.<br />
<br />
<pre><br />
cd /mnt/data1/<user directory>/var/named<br />
mknod -c 1 1 null<br />
mknod -c 2 3 random<br />
</pre><br />
<br />
== Postfix Problems ==<br />
<br />
This hasn't happened in a long time, probably no longer valid with modern OS's<br />
<br />
Postfix is an alternate MTA - people replace sendmail with it in much the same way that many people also replace sendmail with qmail. The problem is that if you install postfix inside of a jail, by default it will not work properly - the user will see error messages like this:<br />
<br />
incoming mail generates errors like this:<br />
<br />
mail_queue_enter: create file incoming/298266.55902: File too large<br />
<br />
and further, sending mail generates errors like this:<br />
<br />
postdrop: warning: mail_queue_enter: create file maildrop/627930.56676: File too large<br />
<br />
This is very easy to solve. In fact, the FreeBSD welcome email that is sent out by `jailmake` now contains this block of text:<br />
<br />
- if you plan on installing postfix, email us first and ask for a necessary patch.<br />
<br />
So, anytime you see a support email that says anything about postfix not working, paste that line from the welcome email into your response to them, and right away, forward the postfix patch to them in an email.<br />
<br />
Here are the instructions:<br />
<br />
Simply copy the patch into /usr/ports/mail/postfix/files, rename the file to ‘patch-file_limit.c’ and then cd to /usr/ports/mail/postfix and run `make install`.<br />
<br />
The postfix patch is the very first email in the support email box, and we never delete or save it because we always want it there to forward to people. The instructions in the email are very clear, and it solves their problem 100% of the time.<br />
<br />
<br />
== Problems with `find` ==<br />
<br />
There was a bug in the FreeBSD vn-filesystem code. The bug causes the system to crash or hang when someone runs the `find` command inside their vn-backed filesystem (and even though the bug may have been fixed, due to the i/o overhead, we still don't like find).<br />
<br />
Now, this does not always happen - however, any jail machine with more than 20 systems on it will crash every night if every system on it runs the daily periodic script out of their crontab, and updates their locate database with the find command.<br />
<br />
(the locate database is a small, FreeBSD specific database that is populated nightly from a cron job, and is then used to provide fast answers to the `locate` command)<br />
<br />
In normal systems (32-40 jails on them), all that has to be done is make sure nobody runs the daily periodic. This means that the file /etc/crontab, instead of looking like this:<br />
<br />
<pre># do daily/weekly/monthly maintenance<br />
1 3 * * * root periodic daily<br />
15 4 * * 6 root periodic weekly<br />
30 5 1 * * root periodic monthly</pre><br />
<br />
Needs to look like this:<br />
<br />
<pre># DO NOT UNCOMMENT THESE - contact support@johncompanies.com for details<br />
#1 3 * * * root periodic daily<br />
#15 4 * * 6 root periodic weekly<br />
#30 5 1 * * root periodic monthly<br />
## DO NOT UNCOMMENT THESE ^^^^^^^^</pre><br />
<br />
and the problem will generally not occur at all. New filesystem images are always altered in this way, so all systems are set like this.<br />
<br />
However, if you have enough systems running, even casual uses of find, outside of the periodic scripts, can cause the system to crash. We are seeing this on jail13, which crashes about once per week.<br />
<br />
There is not yet any resolution to this problem.<br />
<br />
So, if you have a freebsd system crash, see if anyone has uncommented their daily periodic line - you can see them for everyone by running:<br />
<br />
/bin/sh<br />
<br />
and then running:<br />
<br />
<pre>for f in `df -k | grep vn | awk '{print $6}'` ; do echo `cat $f/etc/crontab | grep "periodic daily"` $f ; done | more</pre><br />
<br />
(all one line, one command)<br />
<br />
And then edit those /etc/crontabs and comment them out again.<br />
<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
* if you are getting disk full messages for a BSD customer, it's fairly safe to clear out their /usr/ports/distfiles dir<br />
* 4.x: ps and top an only be run by root in these jails. Done on purpose:<br />
As for `ps` and `top` - non-root users can run them, just not successfully because we have locked the permissions on /dev/mem and /dev/kmem to be root-readable only. That is why non-root users cannot successfully run `ps` and `top`.<br />
* user quotas do not work on freebsd jails - you cannot set up quotas at all, and that's that.<br />
* You cannot inject a process into a 4.x jail, only HUP running processes. 6.x and onward you can with jexec<br />
* jails see base machine’s uptime/load when running top/w<br />
* if someone is unable to get in- cant ping, etc- see if they were blocked by castle (DoS), see if their ip is on the system (post reboot it was lost cause wasn’t in the rc.conf). preboot should catch that.<br />
* in FreeBSD you cant su to root unless you belong to wheel group – so if you remove your acct and setup a new one, we have to add it to wheel group (add to /etc/group)<br />
* Dmesg from underlying sys is seen in customers dmesg on jail<br />
* Popper process going crazy @ 40% for 10min = someone who leaves mail on server<br />
* Don’t force umounts on 4.x jails – it crashes the machine, generally ok on newer machines (running md)<br />
* Good book for admin http://search.barnesandnoble.com/booksearch/isbnInquiry.asp?userid=t824VyRAYz&isbn=0596005164&itm=2<br />
* Self-sign ssl cert http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#selfcert<br />
* conversation with Glenn about semaphores and pgsql<br />
<pre>SDBoody: hey, these are valid amounts/figures/increments right:<br />
kern.ipc.semmni=1280<br />
kern.ipc.semmns=1280<br />
<br />
gr8feen: probably... I always forget exactly what those are, so I usually have to look them up<br />
SDBoody: semaphores<br />
SDBoody: i took the current 1024 and added 256 to them<br />
SDBoody: need more for pgsql<br />
gr8feen: I meant the mni and mns parts...<br />
gr8feen: some of those are not ones you just want to add to... hang on a sec and I'll look them up..<br />
gr8feen: what's semmsl set to?<br />
SDBoody: kern.ipc.semmsl: 1024<br />
<br />
SDBoody: kern.ipc.msgseg: 2048 <br />
kern.ipc.msgssz: 8 <br />
kern.ipc.msgtql: 40 <br />
kern.ipc.msgmnb: 2048 <br />
kern.ipc.msgmni: 40 <br />
kern.ipc.msgmax: 16384 <br />
kern.ipc.semaem: 16384 <br />
kern.ipc.semvmx: 65534 <br />
kern.ipc.semusz: 152 <br />
kern.ipc.semume: 10 <br />
kern.ipc.semopm: 100 <br />
kern.ipc.semmsl: 1024 <br />
kern.ipc.semmnu: 512 <br />
kern.ipc.semmns: 1024 <br />
kern.ipc.semmni: 1024 <br />
kern.ipc.semmap: 768 <br />
kern.ipc.shm_allow_removed: 0 <br />
kern.ipc.shm_use_phys: 1 <br />
kern.ipc.shmall: 262144 <br />
kern.ipc.shmseg: 256 <br />
kern.ipc.shmmni: 784 <br />
kern.ipc.shmmin: 1 <br />
kern.ipc.shmmax: 536870912 <br />
kern.ipc.maxsockets: 25600 <br />
<br />
gr8feen: ok...msl is max per id, mni is max ids, mns is max number of semaphores... so you probably want something like mns = mni * msl<br />
gr8feen: which one did you run out of?<br />
SDBoody: not sure how to tell- ipcs shows the sems in use add up to 1024<br />
SDBoody: there are 59 entries<br />
gr8feen: I'm assuming you tried to start postgres and it failed?<br />
SDBoody: yes<br />
gr8feen: it should have logged why, somewhere..<br />
gr8feen: if I recall, it'll tell you which one it ran out of<br />
SDBoody: > DETAIL: Failed system call was semget(1, 17, 03600).<br />
<br />
gr8feen: so it wanted an id with 17 semaphores...I'd start by making mns = 17*mni and leave mni and mnl set to what they are now and see what it does<br />
SDBoody: i think mni is plenty high<br />
SDBoody: ok, more reasonable 17408<br />
gr8feen: yeah...just change that one and see how it goes..<br />
SDBoody: and leave mni alone at 1024?<br />
gr8feen: yeah...mni id the max number of ids...but if it's trying to get something like 17 per id, your going to hit mns before you hit anything else<br />
SDBoody: right, but doesn't hurt to have it that high (assuming)<br />
<br />
gr8feen: not really... I think those get allocated out of ram that you cant page out, but it's still such a small amount that it really doesn't matter<br />
SDBoody: looks like that worked, thx!<br />
gr8feen: cool<br />
<br />
see semaphores: ipcs -a -s</pre><br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
== Making new customer VE (vm) ==<br />
<br />
This applies only to new virts >= 4.x<br />
<br />
grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.<br />
<br />
1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
also, ping the IP address.<br />
<br />
2. put CT on whichever partition has more space<br />
df -h<br />
<br />
3. vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> ) <br />
vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@johncompanies.com centos-6-x86_64 LM<br />
<br />
4. copy veid, dir, ip and password to pending customer screen. activate customer<br />
<br />
== Making new customer VE (vemakexxx) ==<br />
<br />
This applies to older virts with old templates. This should probably not be used at all anymore.<br />
<br />
<br />
1. look thru hist for ip<br />
<br />
2. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
<br />
3. put ve on whichever partition has more space<br />
vemakerh9 <veid> <ip> <hostname> <mount> <email> [gb disk]; <256|384|512> <veid><br />
vemakerh9 866 69.55.226.109 ngentu.com /vz1 ayo@ngantu.com,asd@asd.com 5; 256 866<br />
<br />
4. copy (veid), dir, and ip to pending customer screen (pass set to p455agfa)<br />
<br />
<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrateonline/migrate failures (migrate manually) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
You may also see migration failing due to quota issues.<br />
<br />
You can try to resolve by copying any quota file into the file you need:<br />
<br />
cp /var/vzquota/quota.1 /var/vzquota/quota.xxx<br />
<br />
If it complains about quota running you should then be able to stop it<br />
<br />
vzquota off xxxx<br />
<br />
If all else fails, migrate to a new VEID<br />
i.e. 1234 becomes 12341<br />
<br />
<br />
If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
'''Note:''' don't forget to fix /usr/local/sbin/backup.config for new disk location<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
=== Cant change file /etc/sysconfig/network ===<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
=== What template is user using ===<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
=== Edits needed to run java ===<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
=== Set time zone ===<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
=== Failing shm_open calls ===<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
=== Debian can't get on the network ===<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
=== CentOS can't get on the network ===<br />
<br />
When CentOS system was running, but can't get on the network, they probably updated iproute via yum.<br />
<br />
Copy an older version of the iproute rpm to their VPS.<br />
<br />
<pre>cp -p /root/iproute/iproute-2.6.32-23.el6.<i686>.rpm /vz2/private/<1527>/fs/root</pre><br />
<br />
Enter their ve, install rpm, restart network, and prevent yum from updating iproute.<br />
<br />
<pre><br />
ve <1527><br />
rpm -i --force iproute-2.6.32-23.el6.<i686>.rpm<br />
rpm -q iproute<br />
rpm -e <new iproute rpm><br />
service network restart<br />
echo "exclude=iproute" >>/etc/yum/ip-exclude<br />
echo "include=/etc/yum/ip-exclude" >>/etc/yum.conf<br />
</pre><br />
<br />
<br />
<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
----<br />
<br />
* uptime/load that vz users see isn’t what we see- it's their own load<br />
* Vz overwrites resolve.conf on every boot to what’s in ve config file. Also overwrites hostname<br />
* Quotas in vz: set Gid_limit in vz config file and restart ve. Errors leading up to this will be out of files but no inode or space issues. set to 2000 in conf, and restart vz<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
To create a new Linux VPS<br />
<br />
* get an IP address from mgmt screen<br />
* run df -h to figure out which disk has enough space<br />
* run vm with no arguments to get syntax and exact template name<br />
* use col0xxxx for hostname if they don’t give you a hostname<br />
* copy over dir, veid, ip and password to pending customer screen<br />
<br />
vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> )<br />
<br />
== disable ==<br />
To disable a customers Virtuozzo VPS server.<br />
<br />
* stop the server<br />
vzctl stop <veid><br />
* prevent restart on server reboot and user restart via Control Panel<br />
vzctl set <veid> --disabled=yes --offline_management=no --save<br />
<br />
== cancelve ==<br />
When a customer cancels all service.<br />
<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
<br />
display the contents of the vz config file for the <veid><br />
<br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
<br />
List all virtual environments running on this machine.<br />
<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
<br />
show processes running under this <veid>.<br />
<br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
<br />
Restart a <veid>.<br />
<br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
<br />
enter a <veid> as root.<br />
<br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
<br />
execute a single command in the <veid> environment.<br />
<br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem VEID <256|512|768|1024|1280|1536|2048> [burst GB]<br />
adjusts the memory resources for the VE. If this is on VZ ver >4, you will also be prompted to provide a burst ram amount. Sample usage:<br />
<br />
<pre># setmem 21671 1024 4<br />
/usr/sbin/vzctl set 21671 --ram 1024m --swap 3072m --save<br />
</pre><br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>75.80.128.147https://wiki.jcihosting.com/index.php?title=VPS_Management&diff=2237VPS Management2020-06-26T01:56:54Z<p>75.80.128.147: /* Ganeti Install a FreeBSD VPS */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
== Moving a server to a different account in the management system ==<br />
<br />
This is a tough one. You gotta touch a bunch of tables in the mysql database.<br />
<pre><br />
systems<br />
subs<br />
ipmap<br />
</pre><br />
<br />
basically, get the sysid and go off that:<br />
<br />
<pre><br />
update systems set cid='newcid' where sysid=xxxxx<br />
update subs set cid='newcid' where sysid=xxxxx<br />
update ipmap set cid='newcid' where sysid=xxxxx<br />
</pre><br />
<br />
You then need to rename the server disk space to change the old ID to the new one.<br />
<br />
For a jail -<br />
<pre><br />
stopjail <hostname><br />
mv 69.55.2XX.XXX-col0<oldid> 69.55.2XX.XXX-col0<newid><br />
mv 69.55.2XX.XXX-col0<oldid>-DIR 69.55.2XX.XXX-col0<newid>-DIR<br />
vi /usr/local/jail/bin/quad1 [Update the ID in the quad file]<br />
startjail <hostname><br />
</pre><br />
<br />
For a virt -<br />
You will have to <br />
* stop the virt <br />
* rename the directories <br />
* modify the config file /vzconf/<veid>.conf <br />
* rename the virt conf file <br />
* restart the new virt.<br />
<br />
that ought to do it..<br />
<br />
<br />
== Re-enable a customer account in the management system ==<br />
<br />
For example to re-instate col03438, In mysql:<br />
<br />
<pre><br />
show tables ;<br />
select * from customers where cid='col03438' ;<br />
update customers set status=1 where cid='col03438' ;<br />
select * from systems where cid='col03438' ;<br />
update systems set stop_date=NULL where cid='col03438' ;<br />
update systems set cancel_date=NULL where cid='col03438' ;<br />
update systems set status=1 where cid='col03438' ;<br />
select * from subs where cid='col03438' ;<br />
update subs set stop_date=NULL where cid='col03438' ;<br />
update subs set status=1 where cid='col03438' <br />
<br />
</pre><br />
<br />
= Ganeti VPS =<br />
<br />
This is how to create a VPS on the Lamphost ganeti cluster.<br />
I have created scripts for creating a FreeBSD VPS and one for<br />
creating a Linux VPS.<br />
<br />
== Ganeti Install a Linux VPS ==<br />
<br />
To create a linux VPS use<br />
<pre><br />
/root/new <name> <disk size> <memory> <IP Addr> [<OS>]<br />
<br />
( for example /root/new vps.example.com 30 2 69.55.238.10 centos-8 )<br />
</pre><br />
<br />
You need to customize the install by adding a "jci" account and a customer<br />
account (col0XXXX) as well. Make sure the customer acct (col0XXXX) is<br />
in a group to allow sudo.<br />
<br />
== Ganeti Install a FreeBSD VPS ==<br />
<br />
To create a FreeBSD VPS, we have to do a full install. This will <br />
create a new VPS. on gn1, do<br />
<br />
<pre><br />
/root/newfreebsd12 <name> <disk size> <memory> <IP Addr> [<OS>]<br />
</pre><br />
<br />
You will then have to do the install via VNC.<br />
<br />
<pre><br />
gnt-instance info <name> | grep -i connection<br />
</pre><br />
<br />
Use your favorite VNC client (I use Tiger VNC). to connect <br />
<br />
<br><br />
You need to customize the install by adding a "jci" account and a customer <br />
account (col0XXXX) as well. Make sure the customer acct (col0XXXX) is in a <br />
group to allow sudo. <br />
<br />
<br> <br />
You will need to install sudo.<br />
<br />
<pre> <br />
pkg install sudo # install sudo package<br />
visudo # to add wheel account to sudoers<br />
vi /etc/group # add jci and user acct to wheel<br />
</pre><br />
<br />
<br><br />
You will also have to set up a console in /etc/ttys.<br />
<pre><br />
vi /etc/ttys<br />
</pre><br />
<br />
== Ganeti restart a VPS ==<br />
<br />
To shutdown a ganeti VPS<br />
<br />
<pre><br />
gnt-instance reboot <VPSname><br />
</pre><br />
<br />
<br />
== Ganeti shutdown a VPS ==<br />
<br />
To shutdown a ganeti VPS<br />
<br />
<pre><br />
gnt-instance shutdown <VPSname><br />
</pre><br />
<br />
To permanently delete a ganeti VPS<br />
<br />
<pre><br />
gnt-instance remove <VPSname><br />
</pre><br />
<br />
== Ganeti Re-Intsall a VPS ==<br />
<br />
Re-installing the OS will destroy all data on the VPS.<br />
Login to the Ganeti master node (currently gn1).<br />
<br />
To reinstall the OS on a VPS the VPS must be shutdown.<br />
<pre><br />
gnt-instance shutdown <VPSname><br />
</pre><br />
<br />
Do the reinstall.<br />
<pre><br />
gnt-instance reinstall <VPSname><br />
</pre><br />
<br />
Restart the VPS.<br />
<pre><br />
gnt-instance startup <VPSname><br />
</pre><br />
<br />
Now you need to do the customization for the VPS. This is<br />
basically the same as the Dedicated server customization.<br />
<br />
https://69.55.229.8/index.php/New_Signups#Dedicated_Server_Install_Checklist<br />
<br />
== Ganeti login to a VPS ==<br />
<br />
To login to a ganeti VPS<br />
<br />
<pre><br />
gnt-instance console <VPSname><br />
<carriage return><br />
<carriage return><br />
<carriage return><br />
</pre><br />
<br />
Use jci and appropriate Ganeti jci password.<br />
<br />
or you can ssh to the VPS name or IP address.<br />
<br />
= FreeBSD 10.1 VPS (jail5 and jail6) =<br />
<br />
This is information on how to manage the FreeBSD 10.1 VPS servers on Jail5.<br />
<br />
<br />
== Making new customer VPS ==<br />
<pre><br />
virtmake IP[,IP] <CID> hostname ipfw# email {FSP|FB|FS|FM|FP} [size in GB]<br />
</pre><br />
<br />
Note: <br />
For now, CID must be unique in the last 2 digits of the CID.<br />
<br />
Must do a "vm start <CID>" after creating a VPS.<br />
<br />
== Starting VPS ==<br />
To start all vms on the system<br />
<pre><br />
vm start<br />
</pre><br />
<br />
To start a single VPS <br />
<pre><br />
vm start <CID><br />
</pre><br />
<br />
== List running VPS ==<br />
<pre><br />
vm status<br />
</pre><br />
<br />
== Attach to a user VPS console ==<br />
<pre><br />
vm attach <CID><br />
</pre><br />
<br />
<br />
To detach from a console, type tilde control-d (~ ctl-d).<br />
<br />
== Stop a VPS ==<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
== Restart a VPS ==<br />
<pre><br />
vm restart <CID><br />
</pre><br />
<br />
<br />
== Modify a VPS ==<br />
<br />
To modify a VPS config, stop the VPS<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
Mount the VPS data file<br />
<pre><br />
vm mount <CID><br />
<br />
cd /data/vm/private/<CID>/mount<br />
</pre><br />
<br />
Modify the user files necessary.<br />
<br />
Restart the VPS.<br />
<br />
<pre><br />
vm umount <CID><br />
<br />
vm start <CID><br />
</pre><br />
<br />
== Notes on a new FBSD 10.x VPS (jail6) ==<br />
<br />
<pre><br />
#! /bin/sh<br />
#<br />
# new <colo#> <disk><br />
<br />
<br />
memory=1024<br />
<br />
x="$1"<br />
<br />
disk="$2"<br />
if [ "y" == "y$disk" ] ; then<br />
disk=60<br />
fi<br />
<br />
disk2=`expr $disk - 4`<br />
<br />
<br />
vm image provision 04dcdf96-d843-11e6-b131-001e67fcc64f col0$x<br />
# vm configure col0$x<br />
sed -e "s/memory=256M/memory=${memory}M/" /vms/col01114/col01114.conf >/vms/col0$x/col0$x.conf<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
# vm start col0$x<br />
<br />
<br />
exit<br />
<br />
vm console col0$x<br />
<br />
#<br />
# to increase the customer disk<br />
#<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
<br />
## on the customer VPS<br />
<br />
gpart recover vtbd0<br />
gpart show vtbd0<br />
swapoff /dev/vtbd0p3<br />
gpart delete -i 3 vtbd0<br />
gpart show vtbd0<br />
sysctl kern.geom.debugflags=16<br />
gpart resize -i 2 -a 4k -s $disk2\G vtbd0<br />
<br />
growfs /dev/vtbd0p2<br />
<br />
gpart show vtbd0<br />
gpart add -t freebsd-swap -a 4k vtbd0<br />
gpart show vtbd0<br />
swapon /dev/vtbd0p3<br />
<br />
df -h<br />
reboot<br />
<br />
vi /etc/rc.conf<br />
hostname<br />
IP Addr<br />
passwd root<br />
passwd user<br />
clear history<br />
cp /dev/null /var/log/utx.log<br />
cp /dev/null ~root/.history<br />
history -c<br />
<br />
</pre><br />
<br />
= FreeBSD Jails =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Making new customer jail 4.x ==<br />
<br />
1. run [[#js|js]] to figure out which partition, IP to put it on, find an unused vn, and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vnX hostname shorthost quad/safe# ipfw# email [gb disk, default 4]</tt><br />
<br />
jail14# /tmp/jailmake 69.55.226.152 /mnt/data2/69.55.226.152-col00182 vn23 negev.cerastes.org negev 4 182 cerastes@cerastes.org<br />
<br />
<br />
== Making new customer jail 6.x ==<br />
<br />
1. run [[#js|js]] to figure out which gvinum vols are free (and which mnt (disk) each gvinum goes with/to), IP to put it on and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vN[,vN] hostname shorthost quad/safe# ipfw# email</tt><br />
<br />
jail19# jailmake 69.55.236.17 /mnt/data1/69.55.236.17-col01574 v10 uncle-leo.commadev.com uncle-leo 1 1574 lee@commadev.com,lee@gmail.com<br />
<br />
<br />
== Changing an ip for freebsd VPS ==<br />
<br />
*stopjail (hostname)<br />
*on systems using vnfile or mdfile: rename vnfile/mdfile, using new ip<br />
*rename dir using new ip<br />
*Edit quad (make sure to update all lines: directory, vn/mdfile)<br />
*buildsafe (not necessary on systems having only quad1)<br />
*if you're adding an ip not already on the host machine: <tt>ipadd x.x.x.x</tt><br />
*startjail (hostname)<br />
if backups: <tt>mvbackups</tt> ...<br />
*Edit dir/ip in mgmt<br />
*if any rules, update any firewall rules to use new ip (careful not to make sure there arent multiple rules per ipfw#- search by colo#)<br />
*(if customer asks and has domains) update their domains on ns1c<br />
*optional: update ipfw counters on the host server<br />
<br />
<br />
== Rename a gconcat vol ==<br />
<br />
*stop jail, unmount everything<br />
*gconcat stop vol<br />
*gconcat clear /dev/gvinum/vN<br />
*gconcat clear /dev/gvinum/vN+1<br />
*gconcat label -v newname /dev/gvinum/vN /dev/gvinum/vN+1<br />
*bsdlabel /dev/concat/newname<br />
(make sure a partition is not fstype unused, if so change to: 4.2BSD)<br />
<br />
<br />
== Remaking a VPS (on same jail) ==<br />
<br />
# take him down (stopjail)<br />
# make note of host, vn/gvin/md, ip<br />
# remove from quad/safe<br />
# move vnfile (rename)<br />
# remove dir (jailmake will make a new one)<br />
# remove ipfw counts<br />
# jailmake<br />
# update backup script<br />
# update db with new dir/ip (use the “move” link, mark old as stopped today)<br />
# update firewall if changed ip<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
<br />
== Adding disk to a 4.x jail ==<br />
<br />
NOTE: if you unmount before vnconfig, re-vnconfig then unmount then vnunconfig<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them.<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to dump to a different partition and create the new vn on the current partition. If no space to make make dump, then move old vn’s to backup or worse, dump to backup.<br />
<br />
<br />
2. <tt>g <customerID></tt> to get the info<br />
<br />
<br />
3. make vnfile, newfs and mount. USE CAUTION not to overwrite existing vn. <br><br />
Ex: <tt>vnconfig –T -S 9g -s labels -c /dev/vn22 /mnt/data2/65.214.160.117-col00496<br><br />
disklabel -r -w vn30 auto<br><br />
newfs /dev/vn30c<br><br />
mkdir /mnt/data2/65.214.160.117-col00496-DIR<br><br />
mount /dev/vn30c /mnt/data2/69.55.237.26-col00241-DIR<br><br />
cd /mnt/data2/65.214.160.117-col00496-DIR</tt><br />
<br />
<br />
4. <tt>jailkill <hostname></tt><br />
<br />
<br />
5. <tt>dump -0a -f - /dev/vn1 | restore -r -f -</tt> <br />
<br />
<br />
6. <tt>rm restoresymtable</tt><br />
<br />
<br />
7. unmount and unvnconfig old system:<br><br />
<tt>umount /dev/vnNNc <br><br />
vnconfig –u /dev/vnNN</tt><br />
<br />
<br />
8. move vnfile. Ex: <tt>mv /mnt/data1/69.55.237.26-col00241 /mnt/data1/old-col00241-vnfile-20110312-noarchive</tt><br />
<br />
<br />
9. remove old dir: <tt>rmdir /mnt/data1/69.55.237.26-col00241-DIR</tt><br />
<br />
<br />
10. edit quad/safe to point to new location, run <tt>[[#buildsafe|buildsafe]]</tt><br />
<br />
<br />
11. start jail: <tt>[[#startjail|startjail]] <hostname></tt><br />
<br />
<br />
12. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
13. mv backups if necessary<br />
<br />
WARNING: if you restore to unmounted vn then you’re actually restoring to /mnt/data1<br />
you can tar and do above to recover<br />
<br />
NOTE: you can’t move a customer to a system that has a base sys that is diff from the one they came from (BSD 4.8 to 4.5 = won’t work)<br />
<br />
NOTE: we don’t mount procfs's to anyone’s system by default.<br />
<br />
NOTE: mount_nulls: <br />
also goofy mount_nulls seen in df -k on jail2<br><br />
also needs to be done in safe and quad<br><br />
if he wants another, use safe/quad as example and add it to file. also exec command at prompt to add mount (no reboot, remount or rejail necessary)<br />
<br />
== Moving customer to another jail machine ==<br />
<br />
Systems should only be moved between similarly-versioned jails. If you cannot get a match, you should go to freebsd.org and see what the differences were between the version you're moving from and the new version. Many of the updates will be to drivers and kernels - these don't affect the customer. Perhaps they updated traceroute or a library...you should copy the new/changed files into the VPS/jail on the new host. An alternate method for moving to a different version would be to buildworld to bring their full distribution up to date. Optional steps for this kind of upgrade are included below.<br />
<br />
1. <tt>g <customerID></tt><br />
<br />
<br />
2. <tt>[[#jailkill|jailkill]] <hostname></tt><br />
<br />
<br />
3. create new device on target system<br><br />
4.x:<br><br />
<pre>vnconfig –T –S 4g –s labels –c /dev/vn1 /mnt/data1/69.55.22x.x-col00XXX<br />
disklabel –r –w vn1 auto<br />
newfs /dev/vn1c</pre><br />
<br />
6.x:<br><br />
<pre>bsdlabel -r -w /dev/gvinum/v1<br />
newfs /dev/gvinum/v1a<br />
or <br />
gconcat label v1-v3 /dev/gvinum/v1 /dev/gvinum/v2 /dev/gvinum/v3<br />
bsdlabel -r -w /dev/concat/v1-v3<br />
newfs /dev/concat/v1-v3a</pre><br />
<br />
7.x+:<br><br />
Run jailmakeempty then skip step 4 & 7 below<br />
<br />
<br />
4. make and mount the dir on new system. <br><br />
Ex: <pre>mkdir /mnt/data2/69.55.230.3-col00123-DIR<br />
chmod 755 /mnt/data2/69.55.230.3-col00123-DIR<br />
mount [device] /mnt/data2/69.55.230.3-col00123-DIR</pre><br />
<br />
<br />
5. [[#stopjail|stopjail]] <hostname> 1<br />
<br />
<br />
6. dump fs to new system. ex: <tt>[[#dumpremoterestore|dumpremoterestore]] /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR</tt> (make sure you can ssh as root on remote machine) <br />
<br />
<br />
6a. OPTIONAL BUILDWORLD:<br><br />
cd /usr/src<br />
make world DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
cd etc<br />
make distribution DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
(you may have to rm an openssh file and re-make dist)<br />
<br />
rm -rf /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/periodic/daily/400.status-disks<br />
vi /etc/periodic/security/100.chksetuid<br />
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt><br><br />
with: <tt>MP='/' (use single quotes)</tt><br />
<br />
Ask user if they want ports overwritten with current, if yes:<br />
cp -r /usr/ports /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/usr<br />
<br />
on source: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d .<br />
(copy in all info)<br />
<br />
on target: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d . <br />
(paste all info)<br />
<br />
on source: <br />
cat /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group<br />
(copy in all info)<br />
<br />
on target:<br />
cat > /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group <br />
(paste all info)<br />
<br />
<br />
7. edit quad on source system copy over entries for the jail to the target system <br />
take care that the vn/gvinum/gconcat/md devices on the target system are’t in use and that the /mnt/dataN path doesn’t need changing<br />
<br />
<br />
8. run [[#buildsafe|buildsafe]] (if this is <=6.x) on target system (copies changes made to quad into safe file)<br />
<br />
<br />
9. remove ip from source system. Ex: <tt>ipdel 69.55.230.3</tt><br />
<br />
<br />
10. add ip to target system. Ex: <tt>ipadd 69.55.230.3</tt><br />
<br />
<br />
11. start new system, with [[#startjail|startjail]] or manually, pasting entries found from running <tt>g <customerID></tt> on the new system<br />
<br />
<br />
12. run <tt>[[#canceljail|canceljail]] col0xxxx</tt> on source system. This should prompt you to remove backups if any existed. Do not let it do this, rather:<br />
<br />
<br />
13. if backups existed move them to the new host via the <tt>[[#mvbackups_.28freebsd.29|mvbackups]] script</tt><br />
<br />
<br />
14. edit quad on source system, edit comment to reflect a move rather than a cancel, ex: <tt># moved to jail2 col00241</tt><br />
<br />
<br />
15. edit mgmt to reflect new host and dir for new system<br />
<br />
<br />
16. optional add ipfw rules on new system. Ex:<br />
<tt>ipfw add 01231 count ip from 69.55.230.2 to any<br />
ipfw add 01232 count ip from any to 69.55.230.2</tt><br />
<br />
== Increasing inodes for a VPS ==<br />
<br />
when doing the newfs: <br />
newfs -i 4096 ...<br />
<br />
<br />
== Jail services unable to contact services within same jail ==<br />
<br />
This is due to a messed up routing table (only seen it on jail2 before). <br />
<br />
route delete 69.55.228.65/32 route add -net 69.55.228.65/32 -iface fxp0 -nostatic -cloning route add 69.55.228.65 -iface lo0<br />
<br />
<br />
== /dev/null permission resets == <br />
<br />
Applies to 4.x jails only<br />
<br />
For some reason that I do not understand at all, the /dev/null node in customer jails, after the system is restarted, often reverts permissions to 0600 ... which is bad, because non-root processes cannot redirect to /dev/null anymore. In fact, a fair number of server daemons will complain and/or fail if this is the case.<br />
<br />
So, after a restart, after all systems have restarted, run postboot, which does something like:<br />
<br />
for f in `df -k | grep /dev/vn | awk '{print $6}'` ; do chmod 0666 $f/dev/null ; done<br />
<br />
Later, after the system is up, if anyone ever complains about their /dev/null permissions for any reason ... really if anyone mentions<br />
/dev/null in any way, tell them to run:<br />
<br />
chmod 0666 /dev/null<br />
<br />
We have ultimately fixed this by adding chmods to the quad/safe's<br />
<br />
<br />
== mknod in a jail == <br />
<br />
For a customer who wants to run a chroot name server (named) in a jail.<br />
<br />
<pre><br />
cd /mnt/data1/<user directory>/var/named<br />
mknod -c 1 1 null<br />
mknod -c 2 3 random<br />
</pre><br />
<br />
== Postfix Problems ==<br />
<br />
This hasn't happened in a long time, probably no longer valid with modern OS's<br />
<br />
Postfix is an alternate MTA - people replace sendmail with it in much the same way that many people also replace sendmail with qmail. The problem is that if you install postfix inside of a jail, by default it will not work properly - the user will see error messages like this:<br />
<br />
incoming mail generates errors like this:<br />
<br />
mail_queue_enter: create file incoming/298266.55902: File too large<br />
<br />
and further, sending mail generates errors like this:<br />
<br />
postdrop: warning: mail_queue_enter: create file maildrop/627930.56676: File too large<br />
<br />
This is very easy to solve. In fact, the FreeBSD welcome email that is sent out by `jailmake` now contains this block of text:<br />
<br />
- if you plan on installing postfix, email us first and ask for a necessary patch.<br />
<br />
So, anytime you see a support email that says anything about postfix not working, paste that line from the welcome email into your response to them, and right away, forward the postfix patch to them in an email.<br />
<br />
Here are the instructions:<br />
<br />
Simply copy the patch into /usr/ports/mail/postfix/files, rename the file to ‘patch-file_limit.c’ and then cd to /usr/ports/mail/postfix and run `make install`.<br />
<br />
The postfix patch is the very first email in the support email box, and we never delete or save it because we always want it there to forward to people. The instructions in the email are very clear, and it solves their problem 100% of the time.<br />
<br />
<br />
== Problems with `find` ==<br />
<br />
There was a bug in the FreeBSD vn-filesystem code. The bug causes the system to crash or hang when someone runs the `find` command inside their vn-backed filesystem (and even though the bug may have been fixed, due to the i/o overhead, we still don't like find).<br />
<br />
Now, this does not always happen - however, any jail machine with more than 20 systems on it will crash every night if every system on it runs the daily periodic script out of their crontab, and updates their locate database with the find command.<br />
<br />
(the locate database is a small, FreeBSD specific database that is populated nightly from a cron job, and is then used to provide fast answers to the `locate` command)<br />
<br />
In normal systems (32-40 jails on them), all that has to be done is make sure nobody runs the daily periodic. This means that the file /etc/crontab, instead of looking like this:<br />
<br />
<pre># do daily/weekly/monthly maintenance<br />
1 3 * * * root periodic daily<br />
15 4 * * 6 root periodic weekly<br />
30 5 1 * * root periodic monthly</pre><br />
<br />
Needs to look like this:<br />
<br />
<pre># DO NOT UNCOMMENT THESE - contact support@johncompanies.com for details<br />
#1 3 * * * root periodic daily<br />
#15 4 * * 6 root periodic weekly<br />
#30 5 1 * * root periodic monthly<br />
## DO NOT UNCOMMENT THESE ^^^^^^^^</pre><br />
<br />
and the problem will generally not occur at all. New filesystem images are always altered in this way, so all systems are set like this.<br />
<br />
However, if you have enough systems running, even casual uses of find, outside of the periodic scripts, can cause the system to crash. We are seeing this on jail13, which crashes about once per week.<br />
<br />
There is not yet any resolution to this problem.<br />
<br />
So, if you have a freebsd system crash, see if anyone has uncommented their daily periodic line - you can see them for everyone by running:<br />
<br />
/bin/sh<br />
<br />
and then running:<br />
<br />
<pre>for f in `df -k | grep vn | awk '{print $6}'` ; do echo `cat $f/etc/crontab | grep "periodic daily"` $f ; done | more</pre><br />
<br />
(all one line, one command)<br />
<br />
And then edit those /etc/crontabs and comment them out again.<br />
<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
* if you are getting disk full messages for a BSD customer, it's fairly safe to clear out their /usr/ports/distfiles dir<br />
* 4.x: ps and top an only be run by root in these jails. Done on purpose:<br />
As for `ps` and `top` - non-root users can run them, just not successfully because we have locked the permissions on /dev/mem and /dev/kmem to be root-readable only. That is why non-root users cannot successfully run `ps` and `top`.<br />
* user quotas do not work on freebsd jails - you cannot set up quotas at all, and that's that.<br />
* You cannot inject a process into a 4.x jail, only HUP running processes. 6.x and onward you can with jexec<br />
* jails see base machine’s uptime/load when running top/w<br />
* if someone is unable to get in- cant ping, etc- see if they were blocked by castle (DoS), see if their ip is on the system (post reboot it was lost cause wasn’t in the rc.conf). preboot should catch that.<br />
* in FreeBSD you cant su to root unless you belong to wheel group – so if you remove your acct and setup a new one, we have to add it to wheel group (add to /etc/group)<br />
* Dmesg from underlying sys is seen in customers dmesg on jail<br />
* Popper process going crazy @ 40% for 10min = someone who leaves mail on server<br />
* Don’t force umounts on 4.x jails – it crashes the machine, generally ok on newer machines (running md)<br />
* Good book for admin http://search.barnesandnoble.com/booksearch/isbnInquiry.asp?userid=t824VyRAYz&isbn=0596005164&itm=2<br />
* Self-sign ssl cert http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#selfcert<br />
* conversation with Glenn about semaphores and pgsql<br />
<pre>SDBoody: hey, these are valid amounts/figures/increments right:<br />
kern.ipc.semmni=1280<br />
kern.ipc.semmns=1280<br />
<br />
gr8feen: probably... I always forget exactly what those are, so I usually have to look them up<br />
SDBoody: semaphores<br />
SDBoody: i took the current 1024 and added 256 to them<br />
SDBoody: need more for pgsql<br />
gr8feen: I meant the mni and mns parts...<br />
gr8feen: some of those are not ones you just want to add to... hang on a sec and I'll look them up..<br />
gr8feen: what's semmsl set to?<br />
SDBoody: kern.ipc.semmsl: 1024<br />
<br />
SDBoody: kern.ipc.msgseg: 2048 <br />
kern.ipc.msgssz: 8 <br />
kern.ipc.msgtql: 40 <br />
kern.ipc.msgmnb: 2048 <br />
kern.ipc.msgmni: 40 <br />
kern.ipc.msgmax: 16384 <br />
kern.ipc.semaem: 16384 <br />
kern.ipc.semvmx: 65534 <br />
kern.ipc.semusz: 152 <br />
kern.ipc.semume: 10 <br />
kern.ipc.semopm: 100 <br />
kern.ipc.semmsl: 1024 <br />
kern.ipc.semmnu: 512 <br />
kern.ipc.semmns: 1024 <br />
kern.ipc.semmni: 1024 <br />
kern.ipc.semmap: 768 <br />
kern.ipc.shm_allow_removed: 0 <br />
kern.ipc.shm_use_phys: 1 <br />
kern.ipc.shmall: 262144 <br />
kern.ipc.shmseg: 256 <br />
kern.ipc.shmmni: 784 <br />
kern.ipc.shmmin: 1 <br />
kern.ipc.shmmax: 536870912 <br />
kern.ipc.maxsockets: 25600 <br />
<br />
gr8feen: ok...msl is max per id, mni is max ids, mns is max number of semaphores... so you probably want something like mns = mni * msl<br />
gr8feen: which one did you run out of?<br />
SDBoody: not sure how to tell- ipcs shows the sems in use add up to 1024<br />
SDBoody: there are 59 entries<br />
gr8feen: I'm assuming you tried to start postgres and it failed?<br />
SDBoody: yes<br />
gr8feen: it should have logged why, somewhere..<br />
gr8feen: if I recall, it'll tell you which one it ran out of<br />
SDBoody: > DETAIL: Failed system call was semget(1, 17, 03600).<br />
<br />
gr8feen: so it wanted an id with 17 semaphores...I'd start by making mns = 17*mni and leave mni and mnl set to what they are now and see what it does<br />
SDBoody: i think mni is plenty high<br />
SDBoody: ok, more reasonable 17408<br />
gr8feen: yeah...just change that one and see how it goes..<br />
SDBoody: and leave mni alone at 1024?<br />
gr8feen: yeah...mni id the max number of ids...but if it's trying to get something like 17 per id, your going to hit mns before you hit anything else<br />
SDBoody: right, but doesn't hurt to have it that high (assuming)<br />
<br />
gr8feen: not really... I think those get allocated out of ram that you cant page out, but it's still such a small amount that it really doesn't matter<br />
SDBoody: looks like that worked, thx!<br />
gr8feen: cool<br />
<br />
see semaphores: ipcs -a -s</pre><br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
== Making new customer VE (vm) ==<br />
<br />
This applies only to new virts >= 4.x<br />
<br />
grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.<br />
<br />
1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
also, ping the IP address.<br />
<br />
2. put CT on whichever partition has more space<br />
df -h<br />
<br />
3. vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> ) <br />
vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@johncompanies.com centos-6-x86_64 LM<br />
<br />
4. copy veid, dir, ip and password to pending customer screen. activate customer<br />
<br />
== Making new customer VE (vemakexxx) ==<br />
<br />
This applies to older virts with old templates. This should probably not be used at all anymore.<br />
<br />
<br />
1. look thru hist for ip<br />
<br />
2. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
<br />
3. put ve on whichever partition has more space<br />
vemakerh9 <veid> <ip> <hostname> <mount> <email> [gb disk]; <256|384|512> <veid><br />
vemakerh9 866 69.55.226.109 ngentu.com /vz1 ayo@ngantu.com,asd@asd.com 5; 256 866<br />
<br />
4. copy (veid), dir, and ip to pending customer screen (pass set to p455agfa)<br />
<br />
<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrateonline/migrate failures (migrate manually) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
You may also see migration failing due to quota issues.<br />
<br />
You can try to resolve by copying any quota file into the file you need:<br />
<br />
cp /var/vzquota/quota.1 /var/vzquota/quota.xxx<br />
<br />
If it complains about quota running you should then be able to stop it<br />
<br />
vzquota off xxxx<br />
<br />
If all else fails, migrate to a new VEID<br />
i.e. 1234 becomes 12341<br />
<br />
<br />
If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
'''Note:''' don't forget to fix /usr/local/sbin/backup.config for new disk location<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
=== Cant change file /etc/sysconfig/network ===<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
=== What template is user using ===<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
=== Edits needed to run java ===<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
=== Set time zone ===<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
=== Failing shm_open calls ===<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
=== Debian can't get on the network ===<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
=== CentOS can't get on the network ===<br />
<br />
When CentOS system was running, but can't get on the network, they probably updated iproute via yum.<br />
<br />
Copy an older version of the iproute rpm to their VPS.<br />
<br />
<pre>cp -p /root/iproute/iproute-2.6.32-23.el6.<i686>.rpm /vz2/private/<1527>/fs/root</pre><br />
<br />
Enter their ve, install rpm, restart network, and prevent yum from updating iproute.<br />
<br />
<pre><br />
ve <1527><br />
rpm -i --force iproute-2.6.32-23.el6.<i686>.rpm<br />
rpm -q iproute<br />
rpm -e <new iproute rpm><br />
service network restart<br />
echo "exclude=iproute" >>/etc/yum/ip-exclude<br />
echo "include=/etc/yum/ip-exclude" >>/etc/yum.conf<br />
</pre><br />
<br />
<br />
<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
----<br />
<br />
* uptime/load that vz users see isn’t what we see- it's their own load<br />
* Vz overwrites resolve.conf on every boot to what’s in ve config file. Also overwrites hostname<br />
* Quotas in vz: set Gid_limit in vz config file and restart ve. Errors leading up to this will be out of files but no inode or space issues. set to 2000 in conf, and restart vz<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
To create a new Linux VPS<br />
<br />
* get an IP address from mgmt screen<br />
* run df -h to figure out which disk has enough space<br />
* run vm with no arguments to get syntax and exact template name<br />
* use col0xxxx for hostname if they don’t give you a hostname<br />
* copy over dir, veid, ip and password to pending customer screen<br />
<br />
vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> )<br />
<br />
== disable ==<br />
To disable a customers Virtuozzo VPS server.<br />
<br />
* stop the server<br />
vzctl stop <veid><br />
* prevent restart on server reboot and user restart via Control Panel<br />
vzctl set <veid> --disabled=yes --offline_management=no --save<br />
<br />
== cancelve ==<br />
When a customer cancels all service.<br />
<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
<br />
display the contents of the vz config file for the <veid><br />
<br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
<br />
List all virtual environments running on this machine.<br />
<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
<br />
show processes running under this <veid>.<br />
<br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
<br />
Restart a <veid>.<br />
<br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
<br />
enter a <veid> as root.<br />
<br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
<br />
execute a single command in the <veid> environment.<br />
<br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem VEID <256|512|768|1024|1280|1536|2048> [burst GB]<br />
adjusts the memory resources for the VE. If this is on VZ ver >4, you will also be prompted to provide a burst ram amount. Sample usage:<br />
<br />
<pre># setmem 21671 1024 4<br />
/usr/sbin/vzctl set 21671 --ram 1024m --swap 3072m --save<br />
</pre><br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>75.80.128.147https://wiki.jcihosting.com/index.php?title=VPS_Management&diff=2236VPS Management2020-06-18T07:14:34Z<p>75.80.128.147: /* Ganeti Re-Intsall a VPS */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
== Moving a server to a different account in the management system ==<br />
<br />
This is a tough one. You gotta touch a bunch of tables in the mysql database.<br />
<pre><br />
systems<br />
subs<br />
ipmap<br />
</pre><br />
<br />
basically, get the sysid and go off that:<br />
<br />
<pre><br />
update systems set cid='newcid' where sysid=xxxxx<br />
update subs set cid='newcid' where sysid=xxxxx<br />
update ipmap set cid='newcid' where sysid=xxxxx<br />
</pre><br />
<br />
You then need to rename the server disk space to change the old ID to the new one.<br />
<br />
For a jail -<br />
<pre><br />
stopjail <hostname><br />
mv 69.55.2XX.XXX-col0<oldid> 69.55.2XX.XXX-col0<newid><br />
mv 69.55.2XX.XXX-col0<oldid>-DIR 69.55.2XX.XXX-col0<newid>-DIR<br />
vi /usr/local/jail/bin/quad1 [Update the ID in the quad file]<br />
startjail <hostname><br />
</pre><br />
<br />
For a virt -<br />
You will have to <br />
* stop the virt <br />
* rename the directories <br />
* modify the config file /vzconf/<veid>.conf <br />
* rename the virt conf file <br />
* restart the new virt.<br />
<br />
that ought to do it..<br />
<br />
<br />
== Re-enable a customer account in the management system ==<br />
<br />
For example to re-instate col03438, In mysql:<br />
<br />
<pre><br />
show tables ;<br />
select * from customers where cid='col03438' ;<br />
update customers set status=1 where cid='col03438' ;<br />
select * from systems where cid='col03438' ;<br />
update systems set stop_date=NULL where cid='col03438' ;<br />
update systems set cancel_date=NULL where cid='col03438' ;<br />
update systems set status=1 where cid='col03438' ;<br />
select * from subs where cid='col03438' ;<br />
update subs set stop_date=NULL where cid='col03438' ;<br />
update subs set status=1 where cid='col03438' <br />
<br />
</pre><br />
<br />
= Ganeti VPS =<br />
<br />
This is how to create a VPS on the Lamphost ganeti cluster.<br />
I have created scripts for creating a FreeBSD VPS and one for<br />
creating a Linux VPS.<br />
<br />
== Ganeti Install a Linux VPS ==<br />
<br />
To create a linux VPS use<br />
<pre><br />
/root/new <name> <disk size> <memory> <IP Addr> [<OS>]<br />
<br />
( for example /root/new vps.example.com 30 2 69.55.238.10 centos-8 )<br />
</pre><br />
<br />
You need to customize the install by adding a "jci" account and a customer<br />
account (col0XXXX) as well. Make sure the customer acct (col0XXXX) is<br />
in a group to allow sudo.<br />
<br />
== Ganeti Install a FreeBSD VPS ==<br />
<br />
To create a FreeBSD VPS, we have to do a full install. This will <br />
create a new VPS.<br />
<br />
<pre><br />
/root/newfreebsd12 <name> <disk size> <memory> <IP Addr> [<OS>]<br />
</pre><br />
<br />
You will then have to do the install via VNC.<br />
<br />
<br><br />
You need to customize the install by adding a "jci" account and a customer <br />
account (col0XXXX) as well. Make sure the customer acct (col0XXXX) is in a <br />
group to allow sudo. <br />
<br />
<br> <br />
You will need to install sudo.<br />
<br />
<pre> <br />
pkg install sudo # install sudo package<br />
visudo # to add wheel account to sudoers<br />
vi /etc/group # add jci and user acct to wheel<br />
</pre><br />
<br />
<br><br />
You will also have to set up a console in /etc/ttys.<br />
<pre><br />
vi /etc/ttys<br />
</pre><br />
<br />
== Ganeti restart a VPS ==<br />
<br />
To shutdown a ganeti VPS<br />
<br />
<pre><br />
gnt-instance reboot <VPSname><br />
</pre><br />
<br />
<br />
== Ganeti shutdown a VPS ==<br />
<br />
To shutdown a ganeti VPS<br />
<br />
<pre><br />
gnt-instance shutdown <VPSname><br />
</pre><br />
<br />
To permanently delete a ganeti VPS<br />
<br />
<pre><br />
gnt-instance remove <VPSname><br />
</pre><br />
<br />
== Ganeti Re-Intsall a VPS ==<br />
<br />
Re-installing the OS will destroy all data on the VPS.<br />
Login to the Ganeti master node (currently gn1).<br />
<br />
To reinstall the OS on a VPS the VPS must be shutdown.<br />
<pre><br />
gnt-instance shutdown <VPSname><br />
</pre><br />
<br />
Do the reinstall.<br />
<pre><br />
gnt-instance reinstall <VPSname><br />
</pre><br />
<br />
Restart the VPS.<br />
<pre><br />
gnt-instance startup <VPSname><br />
</pre><br />
<br />
Now you need to do the customization for the VPS. This is<br />
basically the same as the Dedicated server customization.<br />
<br />
https://69.55.229.8/index.php/New_Signups#Dedicated_Server_Install_Checklist<br />
<br />
== Ganeti login to a VPS ==<br />
<br />
To login to a ganeti VPS<br />
<br />
<pre><br />
gnt-instance console <VPSname><br />
<carriage return><br />
<carriage return><br />
<carriage return><br />
</pre><br />
<br />
Use jci and appropriate Ganeti jci password.<br />
<br />
or you can ssh to the VPS name or IP address.<br />
<br />
= FreeBSD 10.1 VPS (jail5 and jail6) =<br />
<br />
This is information on how to manage the FreeBSD 10.1 VPS servers on Jail5.<br />
<br />
<br />
== Making new customer VPS ==<br />
<pre><br />
virtmake IP[,IP] <CID> hostname ipfw# email {FSP|FB|FS|FM|FP} [size in GB]<br />
</pre><br />
<br />
Note: <br />
For now, CID must be unique in the last 2 digits of the CID.<br />
<br />
Must do a "vm start <CID>" after creating a VPS.<br />
<br />
== Starting VPS ==<br />
To start all vms on the system<br />
<pre><br />
vm start<br />
</pre><br />
<br />
To start a single VPS <br />
<pre><br />
vm start <CID><br />
</pre><br />
<br />
== List running VPS ==<br />
<pre><br />
vm status<br />
</pre><br />
<br />
== Attach to a user VPS console ==<br />
<pre><br />
vm attach <CID><br />
</pre><br />
<br />
<br />
To detach from a console, type tilde control-d (~ ctl-d).<br />
<br />
== Stop a VPS ==<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
== Restart a VPS ==<br />
<pre><br />
vm restart <CID><br />
</pre><br />
<br />
<br />
== Modify a VPS ==<br />
<br />
To modify a VPS config, stop the VPS<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
Mount the VPS data file<br />
<pre><br />
vm mount <CID><br />
<br />
cd /data/vm/private/<CID>/mount<br />
</pre><br />
<br />
Modify the user files necessary.<br />
<br />
Restart the VPS.<br />
<br />
<pre><br />
vm umount <CID><br />
<br />
vm start <CID><br />
</pre><br />
<br />
== Notes on a new FBSD 10.x VPS (jail6) ==<br />
<br />
<pre><br />
#! /bin/sh<br />
#<br />
# new <colo#> <disk><br />
<br />
<br />
memory=1024<br />
<br />
x="$1"<br />
<br />
disk="$2"<br />
if [ "y" == "y$disk" ] ; then<br />
disk=60<br />
fi<br />
<br />
disk2=`expr $disk - 4`<br />
<br />
<br />
vm image provision 04dcdf96-d843-11e6-b131-001e67fcc64f col0$x<br />
# vm configure col0$x<br />
sed -e "s/memory=256M/memory=${memory}M/" /vms/col01114/col01114.conf >/vms/col0$x/col0$x.conf<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
# vm start col0$x<br />
<br />
<br />
exit<br />
<br />
vm console col0$x<br />
<br />
#<br />
# to increase the customer disk<br />
#<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
<br />
## on the customer VPS<br />
<br />
gpart recover vtbd0<br />
gpart show vtbd0<br />
swapoff /dev/vtbd0p3<br />
gpart delete -i 3 vtbd0<br />
gpart show vtbd0<br />
sysctl kern.geom.debugflags=16<br />
gpart resize -i 2 -a 4k -s $disk2\G vtbd0<br />
<br />
growfs /dev/vtbd0p2<br />
<br />
gpart show vtbd0<br />
gpart add -t freebsd-swap -a 4k vtbd0<br />
gpart show vtbd0<br />
swapon /dev/vtbd0p3<br />
<br />
df -h<br />
reboot<br />
<br />
vi /etc/rc.conf<br />
hostname<br />
IP Addr<br />
passwd root<br />
passwd user<br />
clear history<br />
cp /dev/null /var/log/utx.log<br />
cp /dev/null ~root/.history<br />
history -c<br />
<br />
</pre><br />
<br />
= FreeBSD Jails =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Making new customer jail 4.x ==<br />
<br />
1. run [[#js|js]] to figure out which partition, IP to put it on, find an unused vn, and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vnX hostname shorthost quad/safe# ipfw# email [gb disk, default 4]</tt><br />
<br />
jail14# /tmp/jailmake 69.55.226.152 /mnt/data2/69.55.226.152-col00182 vn23 negev.cerastes.org negev 4 182 cerastes@cerastes.org<br />
<br />
<br />
== Making new customer jail 6.x ==<br />
<br />
1. run [[#js|js]] to figure out which gvinum vols are free (and which mnt (disk) each gvinum goes with/to), IP to put it on and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vN[,vN] hostname shorthost quad/safe# ipfw# email</tt><br />
<br />
jail19# jailmake 69.55.236.17 /mnt/data1/69.55.236.17-col01574 v10 uncle-leo.commadev.com uncle-leo 1 1574 lee@commadev.com,lee@gmail.com<br />
<br />
<br />
== Changing an ip for freebsd VPS ==<br />
<br />
*stopjail (hostname)<br />
*on systems using vnfile or mdfile: rename vnfile/mdfile, using new ip<br />
*rename dir using new ip<br />
*Edit quad (make sure to update all lines: directory, vn/mdfile)<br />
*buildsafe (not necessary on systems having only quad1)<br />
*if you're adding an ip not already on the host machine: <tt>ipadd x.x.x.x</tt><br />
*startjail (hostname)<br />
if backups: <tt>mvbackups</tt> ...<br />
*Edit dir/ip in mgmt<br />
*if any rules, update any firewall rules to use new ip (careful not to make sure there arent multiple rules per ipfw#- search by colo#)<br />
*(if customer asks and has domains) update their domains on ns1c<br />
*optional: update ipfw counters on the host server<br />
<br />
<br />
== Rename a gconcat vol ==<br />
<br />
*stop jail, unmount everything<br />
*gconcat stop vol<br />
*gconcat clear /dev/gvinum/vN<br />
*gconcat clear /dev/gvinum/vN+1<br />
*gconcat label -v newname /dev/gvinum/vN /dev/gvinum/vN+1<br />
*bsdlabel /dev/concat/newname<br />
(make sure a partition is not fstype unused, if so change to: 4.2BSD)<br />
<br />
<br />
== Remaking a VPS (on same jail) ==<br />
<br />
# take him down (stopjail)<br />
# make note of host, vn/gvin/md, ip<br />
# remove from quad/safe<br />
# move vnfile (rename)<br />
# remove dir (jailmake will make a new one)<br />
# remove ipfw counts<br />
# jailmake<br />
# update backup script<br />
# update db with new dir/ip (use the “move” link, mark old as stopped today)<br />
# update firewall if changed ip<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
<br />
== Adding disk to a 4.x jail ==<br />
<br />
NOTE: if you unmount before vnconfig, re-vnconfig then unmount then vnunconfig<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them.<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to dump to a different partition and create the new vn on the current partition. If no space to make make dump, then move old vn’s to backup or worse, dump to backup.<br />
<br />
<br />
2. <tt>g <customerID></tt> to get the info<br />
<br />
<br />
3. make vnfile, newfs and mount. USE CAUTION not to overwrite existing vn. <br><br />
Ex: <tt>vnconfig –T -S 9g -s labels -c /dev/vn22 /mnt/data2/65.214.160.117-col00496<br><br />
disklabel -r -w vn30 auto<br><br />
newfs /dev/vn30c<br><br />
mkdir /mnt/data2/65.214.160.117-col00496-DIR<br><br />
mount /dev/vn30c /mnt/data2/69.55.237.26-col00241-DIR<br><br />
cd /mnt/data2/65.214.160.117-col00496-DIR</tt><br />
<br />
<br />
4. <tt>jailkill <hostname></tt><br />
<br />
<br />
5. <tt>dump -0a -f - /dev/vn1 | restore -r -f -</tt> <br />
<br />
<br />
6. <tt>rm restoresymtable</tt><br />
<br />
<br />
7. unmount and unvnconfig old system:<br><br />
<tt>umount /dev/vnNNc <br><br />
vnconfig –u /dev/vnNN</tt><br />
<br />
<br />
8. move vnfile. Ex: <tt>mv /mnt/data1/69.55.237.26-col00241 /mnt/data1/old-col00241-vnfile-20110312-noarchive</tt><br />
<br />
<br />
9. remove old dir: <tt>rmdir /mnt/data1/69.55.237.26-col00241-DIR</tt><br />
<br />
<br />
10. edit quad/safe to point to new location, run <tt>[[#buildsafe|buildsafe]]</tt><br />
<br />
<br />
11. start jail: <tt>[[#startjail|startjail]] <hostname></tt><br />
<br />
<br />
12. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
13. mv backups if necessary<br />
<br />
WARNING: if you restore to unmounted vn then you’re actually restoring to /mnt/data1<br />
you can tar and do above to recover<br />
<br />
NOTE: you can’t move a customer to a system that has a base sys that is diff from the one they came from (BSD 4.8 to 4.5 = won’t work)<br />
<br />
NOTE: we don’t mount procfs's to anyone’s system by default.<br />
<br />
NOTE: mount_nulls: <br />
also goofy mount_nulls seen in df -k on jail2<br><br />
also needs to be done in safe and quad<br><br />
if he wants another, use safe/quad as example and add it to file. also exec command at prompt to add mount (no reboot, remount or rejail necessary)<br />
<br />
== Moving customer to another jail machine ==<br />
<br />
Systems should only be moved between similarly-versioned jails. If you cannot get a match, you should go to freebsd.org and see what the differences were between the version you're moving from and the new version. Many of the updates will be to drivers and kernels - these don't affect the customer. Perhaps they updated traceroute or a library...you should copy the new/changed files into the VPS/jail on the new host. An alternate method for moving to a different version would be to buildworld to bring their full distribution up to date. Optional steps for this kind of upgrade are included below.<br />
<br />
1. <tt>g <customerID></tt><br />
<br />
<br />
2. <tt>[[#jailkill|jailkill]] <hostname></tt><br />
<br />
<br />
3. create new device on target system<br><br />
4.x:<br><br />
<pre>vnconfig –T –S 4g –s labels –c /dev/vn1 /mnt/data1/69.55.22x.x-col00XXX<br />
disklabel –r –w vn1 auto<br />
newfs /dev/vn1c</pre><br />
<br />
6.x:<br><br />
<pre>bsdlabel -r -w /dev/gvinum/v1<br />
newfs /dev/gvinum/v1a<br />
or <br />
gconcat label v1-v3 /dev/gvinum/v1 /dev/gvinum/v2 /dev/gvinum/v3<br />
bsdlabel -r -w /dev/concat/v1-v3<br />
newfs /dev/concat/v1-v3a</pre><br />
<br />
7.x+:<br><br />
Run jailmakeempty then skip step 4 & 7 below<br />
<br />
<br />
4. make and mount the dir on new system. <br><br />
Ex: <pre>mkdir /mnt/data2/69.55.230.3-col00123-DIR<br />
chmod 755 /mnt/data2/69.55.230.3-col00123-DIR<br />
mount [device] /mnt/data2/69.55.230.3-col00123-DIR</pre><br />
<br />
<br />
5. [[#stopjail|stopjail]] <hostname> 1<br />
<br />
<br />
6. dump fs to new system. ex: <tt>[[#dumpremoterestore|dumpremoterestore]] /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR</tt> (make sure you can ssh as root on remote machine) <br />
<br />
<br />
6a. OPTIONAL BUILDWORLD:<br><br />
cd /usr/src<br />
make world DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
cd etc<br />
make distribution DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
(you may have to rm an openssh file and re-make dist)<br />
<br />
rm -rf /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/periodic/daily/400.status-disks<br />
vi /etc/periodic/security/100.chksetuid<br />
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt><br><br />
with: <tt>MP='/' (use single quotes)</tt><br />
<br />
Ask user if they want ports overwritten with current, if yes:<br />
cp -r /usr/ports /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/usr<br />
<br />
on source: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d .<br />
(copy in all info)<br />
<br />
on target: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d . <br />
(paste all info)<br />
<br />
on source: <br />
cat /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group<br />
(copy in all info)<br />
<br />
on target:<br />
cat > /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group <br />
(paste all info)<br />
<br />
<br />
7. edit quad on source system copy over entries for the jail to the target system <br />
take care that the vn/gvinum/gconcat/md devices on the target system are’t in use and that the /mnt/dataN path doesn’t need changing<br />
<br />
<br />
8. run [[#buildsafe|buildsafe]] (if this is <=6.x) on target system (copies changes made to quad into safe file)<br />
<br />
<br />
9. remove ip from source system. Ex: <tt>ipdel 69.55.230.3</tt><br />
<br />
<br />
10. add ip to target system. Ex: <tt>ipadd 69.55.230.3</tt><br />
<br />
<br />
11. start new system, with [[#startjail|startjail]] or manually, pasting entries found from running <tt>g <customerID></tt> on the new system<br />
<br />
<br />
12. run <tt>[[#canceljail|canceljail]] col0xxxx</tt> on source system. This should prompt you to remove backups if any existed. Do not let it do this, rather:<br />
<br />
<br />
13. if backups existed move them to the new host via the <tt>[[#mvbackups_.28freebsd.29|mvbackups]] script</tt><br />
<br />
<br />
14. edit quad on source system, edit comment to reflect a move rather than a cancel, ex: <tt># moved to jail2 col00241</tt><br />
<br />
<br />
15. edit mgmt to reflect new host and dir for new system<br />
<br />
<br />
16. optional add ipfw rules on new system. Ex:<br />
<tt>ipfw add 01231 count ip from 69.55.230.2 to any<br />
ipfw add 01232 count ip from any to 69.55.230.2</tt><br />
<br />
== Increasing inodes for a VPS ==<br />
<br />
when doing the newfs: <br />
newfs -i 4096 ...<br />
<br />
<br />
== Jail services unable to contact services within same jail ==<br />
<br />
This is due to a messed up routing table (only seen it on jail2 before). <br />
<br />
route delete 69.55.228.65/32 route add -net 69.55.228.65/32 -iface fxp0 -nostatic -cloning route add 69.55.228.65 -iface lo0<br />
<br />
<br />
== /dev/null permission resets == <br />
<br />
Applies to 4.x jails only<br />
<br />
For some reason that I do not understand at all, the /dev/null node in customer jails, after the system is restarted, often reverts permissions to 0600 ... which is bad, because non-root processes cannot redirect to /dev/null anymore. In fact, a fair number of server daemons will complain and/or fail if this is the case.<br />
<br />
So, after a restart, after all systems have restarted, run postboot, which does something like:<br />
<br />
for f in `df -k | grep /dev/vn | awk '{print $6}'` ; do chmod 0666 $f/dev/null ; done<br />
<br />
Later, after the system is up, if anyone ever complains about their /dev/null permissions for any reason ... really if anyone mentions<br />
/dev/null in any way, tell them to run:<br />
<br />
chmod 0666 /dev/null<br />
<br />
We have ultimately fixed this by adding chmods to the quad/safe's<br />
<br />
<br />
== mknod in a jail == <br />
<br />
For a customer who wants to run a chroot name server (named) in a jail.<br />
<br />
<pre><br />
cd /mnt/data1/<user directory>/var/named<br />
mknod -c 1 1 null<br />
mknod -c 2 3 random<br />
</pre><br />
<br />
== Postfix Problems ==<br />
<br />
This hasn't happened in a long time, probably no longer valid with modern OS's<br />
<br />
Postfix is an alternate MTA - people replace sendmail with it in much the same way that many people also replace sendmail with qmail. The problem is that if you install postfix inside of a jail, by default it will not work properly - the user will see error messages like this:<br />
<br />
incoming mail generates errors like this:<br />
<br />
mail_queue_enter: create file incoming/298266.55902: File too large<br />
<br />
and further, sending mail generates errors like this:<br />
<br />
postdrop: warning: mail_queue_enter: create file maildrop/627930.56676: File too large<br />
<br />
This is very easy to solve. In fact, the FreeBSD welcome email that is sent out by `jailmake` now contains this block of text:<br />
<br />
- if you plan on installing postfix, email us first and ask for a necessary patch.<br />
<br />
So, anytime you see a support email that says anything about postfix not working, paste that line from the welcome email into your response to them, and right away, forward the postfix patch to them in an email.<br />
<br />
Here are the instructions:<br />
<br />
Simply copy the patch into /usr/ports/mail/postfix/files, rename the file to ‘patch-file_limit.c’ and then cd to /usr/ports/mail/postfix and run `make install`.<br />
<br />
The postfix patch is the very first email in the support email box, and we never delete or save it because we always want it there to forward to people. The instructions in the email are very clear, and it solves their problem 100% of the time.<br />
<br />
<br />
== Problems with `find` ==<br />
<br />
There was a bug in the FreeBSD vn-filesystem code. The bug causes the system to crash or hang when someone runs the `find` command inside their vn-backed filesystem (and even though the bug may have been fixed, due to the i/o overhead, we still don't like find).<br />
<br />
Now, this does not always happen - however, any jail machine with more than 20 systems on it will crash every night if every system on it runs the daily periodic script out of their crontab, and updates their locate database with the find command.<br />
<br />
(the locate database is a small, FreeBSD specific database that is populated nightly from a cron job, and is then used to provide fast answers to the `locate` command)<br />
<br />
In normal systems (32-40 jails on them), all that has to be done is make sure nobody runs the daily periodic. This means that the file /etc/crontab, instead of looking like this:<br />
<br />
<pre># do daily/weekly/monthly maintenance<br />
1 3 * * * root periodic daily<br />
15 4 * * 6 root periodic weekly<br />
30 5 1 * * root periodic monthly</pre><br />
<br />
Needs to look like this:<br />
<br />
<pre># DO NOT UNCOMMENT THESE - contact support@johncompanies.com for details<br />
#1 3 * * * root periodic daily<br />
#15 4 * * 6 root periodic weekly<br />
#30 5 1 * * root periodic monthly<br />
## DO NOT UNCOMMENT THESE ^^^^^^^^</pre><br />
<br />
and the problem will generally not occur at all. New filesystem images are always altered in this way, so all systems are set like this.<br />
<br />
However, if you have enough systems running, even casual uses of find, outside of the periodic scripts, can cause the system to crash. We are seeing this on jail13, which crashes about once per week.<br />
<br />
There is not yet any resolution to this problem.<br />
<br />
So, if you have a freebsd system crash, see if anyone has uncommented their daily periodic line - you can see them for everyone by running:<br />
<br />
/bin/sh<br />
<br />
and then running:<br />
<br />
<pre>for f in `df -k | grep vn | awk '{print $6}'` ; do echo `cat $f/etc/crontab | grep "periodic daily"` $f ; done | more</pre><br />
<br />
(all one line, one command)<br />
<br />
And then edit those /etc/crontabs and comment them out again.<br />
<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
* if you are getting disk full messages for a BSD customer, it's fairly safe to clear out their /usr/ports/distfiles dir<br />
* 4.x: ps and top an only be run by root in these jails. Done on purpose:<br />
As for `ps` and `top` - non-root users can run them, just not successfully because we have locked the permissions on /dev/mem and /dev/kmem to be root-readable only. That is why non-root users cannot successfully run `ps` and `top`.<br />
* user quotas do not work on freebsd jails - you cannot set up quotas at all, and that's that.<br />
* You cannot inject a process into a 4.x jail, only HUP running processes. 6.x and onward you can with jexec<br />
* jails see base machine’s uptime/load when running top/w<br />
* if someone is unable to get in- cant ping, etc- see if they were blocked by castle (DoS), see if their ip is on the system (post reboot it was lost cause wasn’t in the rc.conf). preboot should catch that.<br />
* in FreeBSD you cant su to root unless you belong to wheel group – so if you remove your acct and setup a new one, we have to add it to wheel group (add to /etc/group)<br />
* Dmesg from underlying sys is seen in customers dmesg on jail<br />
* Popper process going crazy @ 40% for 10min = someone who leaves mail on server<br />
* Don’t force umounts on 4.x jails – it crashes the machine, generally ok on newer machines (running md)<br />
* Good book for admin http://search.barnesandnoble.com/booksearch/isbnInquiry.asp?userid=t824VyRAYz&isbn=0596005164&itm=2<br />
* Self-sign ssl cert http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#selfcert<br />
* conversation with Glenn about semaphores and pgsql<br />
<pre>SDBoody: hey, these are valid amounts/figures/increments right:<br />
kern.ipc.semmni=1280<br />
kern.ipc.semmns=1280<br />
<br />
gr8feen: probably... I always forget exactly what those are, so I usually have to look them up<br />
SDBoody: semaphores<br />
SDBoody: i took the current 1024 and added 256 to them<br />
SDBoody: need more for pgsql<br />
gr8feen: I meant the mni and mns parts...<br />
gr8feen: some of those are not ones you just want to add to... hang on a sec and I'll look them up..<br />
gr8feen: what's semmsl set to?<br />
SDBoody: kern.ipc.semmsl: 1024<br />
<br />
SDBoody: kern.ipc.msgseg: 2048 <br />
kern.ipc.msgssz: 8 <br />
kern.ipc.msgtql: 40 <br />
kern.ipc.msgmnb: 2048 <br />
kern.ipc.msgmni: 40 <br />
kern.ipc.msgmax: 16384 <br />
kern.ipc.semaem: 16384 <br />
kern.ipc.semvmx: 65534 <br />
kern.ipc.semusz: 152 <br />
kern.ipc.semume: 10 <br />
kern.ipc.semopm: 100 <br />
kern.ipc.semmsl: 1024 <br />
kern.ipc.semmnu: 512 <br />
kern.ipc.semmns: 1024 <br />
kern.ipc.semmni: 1024 <br />
kern.ipc.semmap: 768 <br />
kern.ipc.shm_allow_removed: 0 <br />
kern.ipc.shm_use_phys: 1 <br />
kern.ipc.shmall: 262144 <br />
kern.ipc.shmseg: 256 <br />
kern.ipc.shmmni: 784 <br />
kern.ipc.shmmin: 1 <br />
kern.ipc.shmmax: 536870912 <br />
kern.ipc.maxsockets: 25600 <br />
<br />
gr8feen: ok...msl is max per id, mni is max ids, mns is max number of semaphores... so you probably want something like mns = mni * msl<br />
gr8feen: which one did you run out of?<br />
SDBoody: not sure how to tell- ipcs shows the sems in use add up to 1024<br />
SDBoody: there are 59 entries<br />
gr8feen: I'm assuming you tried to start postgres and it failed?<br />
SDBoody: yes<br />
gr8feen: it should have logged why, somewhere..<br />
gr8feen: if I recall, it'll tell you which one it ran out of<br />
SDBoody: > DETAIL: Failed system call was semget(1, 17, 03600).<br />
<br />
gr8feen: so it wanted an id with 17 semaphores...I'd start by making mns = 17*mni and leave mni and mnl set to what they are now and see what it does<br />
SDBoody: i think mni is plenty high<br />
SDBoody: ok, more reasonable 17408<br />
gr8feen: yeah...just change that one and see how it goes..<br />
SDBoody: and leave mni alone at 1024?<br />
gr8feen: yeah...mni id the max number of ids...but if it's trying to get something like 17 per id, your going to hit mns before you hit anything else<br />
SDBoody: right, but doesn't hurt to have it that high (assuming)<br />
<br />
gr8feen: not really... I think those get allocated out of ram that you cant page out, but it's still such a small amount that it really doesn't matter<br />
SDBoody: looks like that worked, thx!<br />
gr8feen: cool<br />
<br />
see semaphores: ipcs -a -s</pre><br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
== Making new customer VE (vm) ==<br />
<br />
This applies only to new virts >= 4.x<br />
<br />
grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.<br />
<br />
1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
also, ping the IP address.<br />
<br />
2. put CT on whichever partition has more space<br />
df -h<br />
<br />
3. vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> ) <br />
vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@johncompanies.com centos-6-x86_64 LM<br />
<br />
4. copy veid, dir, ip and password to pending customer screen. activate customer<br />
<br />
== Making new customer VE (vemakexxx) ==<br />
<br />
This applies to older virts with old templates. This should probably not be used at all anymore.<br />
<br />
<br />
1. look thru hist for ip<br />
<br />
2. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
<br />
3. put ve on whichever partition has more space<br />
vemakerh9 <veid> <ip> <hostname> <mount> <email> [gb disk]; <256|384|512> <veid><br />
vemakerh9 866 69.55.226.109 ngentu.com /vz1 ayo@ngantu.com,asd@asd.com 5; 256 866<br />
<br />
4. copy (veid), dir, and ip to pending customer screen (pass set to p455agfa)<br />
<br />
<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrateonline/migrate failures (migrate manually) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
You may also see migration failing due to quota issues.<br />
<br />
You can try to resolve by copying any quota file into the file you need:<br />
<br />
cp /var/vzquota/quota.1 /var/vzquota/quota.xxx<br />
<br />
If it complains about quota running you should then be able to stop it<br />
<br />
vzquota off xxxx<br />
<br />
If all else fails, migrate to a new VEID<br />
i.e. 1234 becomes 12341<br />
<br />
<br />
If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
'''Note:''' don't forget to fix /usr/local/sbin/backup.config for new disk location<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
=== Cant change file /etc/sysconfig/network ===<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
=== What template is user using ===<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
=== Edits needed to run java ===<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
=== Set time zone ===<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
=== Failing shm_open calls ===<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
=== Debian can't get on the network ===<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
=== CentOS can't get on the network ===<br />
<br />
When CentOS system was running, but can't get on the network, they probably updated iproute via yum.<br />
<br />
Copy an older version of the iproute rpm to their VPS.<br />
<br />
<pre>cp -p /root/iproute/iproute-2.6.32-23.el6.<i686>.rpm /vz2/private/<1527>/fs/root</pre><br />
<br />
Enter their ve, install rpm, restart network, and prevent yum from updating iproute.<br />
<br />
<pre><br />
ve <1527><br />
rpm -i --force iproute-2.6.32-23.el6.<i686>.rpm<br />
rpm -q iproute<br />
rpm -e <new iproute rpm><br />
service network restart<br />
echo "exclude=iproute" >>/etc/yum/ip-exclude<br />
echo "include=/etc/yum/ip-exclude" >>/etc/yum.conf<br />
</pre><br />
<br />
<br />
<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
----<br />
<br />
* uptime/load that vz users see isn’t what we see- it's their own load<br />
* Vz overwrites resolve.conf on every boot to what’s in ve config file. Also overwrites hostname<br />
* Quotas in vz: set Gid_limit in vz config file and restart ve. Errors leading up to this will be out of files but no inode or space issues. set to 2000 in conf, and restart vz<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
To create a new Linux VPS<br />
<br />
* get an IP address from mgmt screen<br />
* run df -h to figure out which disk has enough space<br />
* run vm with no arguments to get syntax and exact template name<br />
* use col0xxxx for hostname if they don’t give you a hostname<br />
* copy over dir, veid, ip and password to pending customer screen<br />
<br />
vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> )<br />
<br />
== disable ==<br />
To disable a customers Virtuozzo VPS server.<br />
<br />
* stop the server<br />
vzctl stop <veid><br />
* prevent restart on server reboot and user restart via Control Panel<br />
vzctl set <veid> --disabled=yes --offline_management=no --save<br />
<br />
== cancelve ==<br />
When a customer cancels all service.<br />
<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
<br />
display the contents of the vz config file for the <veid><br />
<br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
<br />
List all virtual environments running on this machine.<br />
<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
<br />
show processes running under this <veid>.<br />
<br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
<br />
Restart a <veid>.<br />
<br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
<br />
enter a <veid> as root.<br />
<br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
<br />
execute a single command in the <veid> environment.<br />
<br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem VEID <256|512|768|1024|1280|1536|2048> [burst GB]<br />
adjusts the memory resources for the VE. If this is on VZ ver >4, you will also be prompted to provide a burst ram amount. Sample usage:<br />
<br />
<pre># setmem 21671 1024 4<br />
/usr/sbin/vzctl set 21671 --ram 1024m --swap 3072m --save<br />
</pre><br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>75.80.128.147https://wiki.jcihosting.com/index.php?title=Routine_Maintenance&diff=2232Routine Maintenance2020-06-09T22:02:19Z<p>75.80.128.147: /* Free up space on Other Servers */</p>
<hr />
<div>= Daily Tasks =<br />
<br />
== check load graphs == <br />
<br />
Click on the Load link in mgmt <br />
<br />
This screen shows you load levels on our servers and network traffic for critical machines (firewalls, backup servers).<br />
<br />
If you see load high or increasing <br />
<br />
FreeBSD: <br />
run [[VPS_Management#jtop|jtop]] (or [[VPS_Management#jt|jt]] > 7.x) and see if there are any runaway processes. Here are some examples of entries in top that are<br />
definitely runaway processes:<br />
<br />
<pre>79481 root 64 0 2256K 1056K CPU1 1 58:16 87.40% 87.40% nano<br />
50650 1000 64 0 1852K 1112K RUN 0 207.9H 84.08% 84.08% screen<br />
14829 www 2 0 39100K 31736K accept 0 104:24 46.54% 6.54% httpd<br />
42065 root 61 0 1300K 844K RUN 1 47.8H 91.36% 91.36% ee<br />
1328 www 56 0 18440K 10796K CPU1 0 64.4H 97.71% 97.71% httpd<br />
26251 user 57 0 6124K 1160K CPU1 1 82.9H 98.44% 98.44% screen<br />
89874 root 60 0 1352K 892K RUN 1 33.8H 65.82% 65.82% dialog<br />
38656 1000 64 0 3088K 2136K CPU0 0 806:13 97.95% 97.95% StutBot<br />
27630 root 64 0 1396K 972K RUN 1 76.8H 86.47% 86.47% ee</pre><br />
<br />
Linux:<br />
run [[VPS_Management#vwe|vwe]] to see which VPS’s have high loads. From there run <tt>[[VPS_Management#vp|vp]] <veid></tt> and/or <tt>[[VPS_Management#vt|vt]] <veid></tt> to see what's going on in that system.<br />
[[VPS_Management#vzstat|vzstat]] will also give you a nice picture of whats going on, systems with high numbers in the mlat column are likely culprits.<br />
<br />
examples of out of control procs:<br />
<br />
<pre>12183 nobody 16 0 4916 1348 1340 R 45.5 0.0 4249m httpd<br />
29266 #502 16 0 1852 796 792 R 22.5 0.0 1104m vim<br />
23860 #41 16 0 5472 5472 2076 R 98.9 0.2 31:41 python<br />
19227 bin 19 0 1688 716 652 R 99.9 0.0 321:08 wtrs_ui<br />
7762 apache 16 0 268 236 224 R 85.7 0.0 1010m ptrace<br />
4624 #501 20 0 4304 2400 2044 R 53.6 0.1 284:32 YoSucker<br />
20451 #506 20 0 1876 820 816 R 17.2 0.0 169:35 vim<br />
8834 #514 20 0 900 724 672 R 77.6 0.0 382:30 neostats<br />
31815 apache 14 0 3176 3176 1696 R 74.4 0.1 6:15 counter</pre><br />
<br />
Just kill -9 them and be done with it.<br />
<br />
Also, anytime you see `kmod` or `ptrace` - kill those immediaely no matter how much they are using - they are attempts to exploit the linux ptrace bug. They won't work, but they suck a lot of CPU...<br />
<br />
Also, any other processes that are at 90-100% cpu usage and have been running for any long period of time should be killed except for mysqld processes on FreeBSD. See above.<br />
<br />
However, there is an exception:<br />
<br />
if it is a mysqld, we don't want to kill their database. What you want to do is <tt>[[VPS_Management#jpid|jpid]] <pid></tt> to see who owns it, and then email them the paste containing the instructions for the nanny. Or you can simply do a <tt>kill -1 PID</tt> on the process to restart it.<br />
<br />
=== Load averages jump at night ===<br />
<br />
The load averages on the FreeBSD systems may jump up at night between 1 and 4 am - this is because the backups are running - if this is what is causing a jump in load, you will see processes like `rsync` in top eating a lot of CPU time.<br />
<br />
== check backups ==<br />
<br />
mgmt -> Motnroing -> Backups and make sure every machine was backed up the previous nite. <br />
Also look at df on backup1 and backup2 to make sure no disk is approaching full, though bb should warn us in advance. Please note - errors encountered when a backup script on any of the particular systems run will gnerate an email to support@johncompanies.com so you can know immediately the day after if the directory to be backed up has been moved or no longer exists. A paste exists for this to notify the customer of a non-existant file/dir.<br />
<br />
== check bb for warnings ==<br />
mgmt -> BigBrother<br />
<br />
Some events don't generate pages (on purpose). You will only see them by going to the bb main page.<br />
<br />
== check jail5 for crashed VPSs ==<br />
<br />
On jail5<br />
<br />
notrunning<br />
<br />
To restart a VPS <br />
<br />
vm restart col0XXXX<br />
<br />
== Check NetHere ==<br />
<br />
Check the NetHere servers. To get into the servers, login to<br />
admin-1.nethere.net and su - to root.<br />
<br />
=== Mail systems ===<br />
Check for possible SPAMMERs.<br />
<br />
==== Incoming ====<br />
Check mta-1 and mta-2 count of customer logins for possible<br />
customer SPAM compromises.<br />
<br />
login_count /logs/maillog | tail -30<br />
<br />
==== Outgoing ====<br />
<br />
Check outgoing queues on relay-1 and relay-2<br />
<br />
mail_count | tail -30<br />
<br />
To clean up outgoing queues of unwanted SPAM on relay-1 and relay-2.<br />
<br />
mail_cleanup [ <sender's domain/username/message id> ... ]<br />
<br />
To just remove emails from some senders.<br />
<br />
rmmails <sender's domain/username/message id> ...<br />
<br />
=== Nagios ===<br />
<br />
Check for other problems on NetHere servers<br />
<br />
https://nagios.nethere.net<br />
<br />
=== Cacti ===<br />
<br />
Check bandwidth usage on servers<br />
<br />
https://cacti.nethere.net<br />
<br />
= Monthly Tasks =<br />
== rotate pine sent mail (1st of month) ==<br />
On the 1st of the month, before any emails are sent out, quit out of pine, then log back in. Send mail from last month will be archived.<br />
If you mess up and do it on the 3rd (for example), you can go into the previous month's saved email and save emails from the current month into the <tt>sent-mail</tt> (current month) mailbox.<br />
<br />
== b/w caps ==<br />
On the 1st: remove any bwcaps put into the firewall (only really applies if a bwcap was added cause someone went over on b/w):<br />
<br />
ipfw list|grep pipe<br />
ipfw del [each rule listed]<br />
<br />
NOTE: this cronjob on newgateway will do some of that for you, provided you used one of the following pipe #s:<br />
0 0 1 * * /sbin/ipfw del 3 4 5 17331<br />
<br />
-------------------<br />
<br />
We really don’t do this anymore since we have centralized traffic accounting with netflow, but for posterity:<br />
<br />
Make sure all machines reset counters to 0 after midnight on the 1st<br />
Make sure they dumped a counter<br />
<br />
On each jail run:<br />
trafficgather.pl<br />
<br />
And on each virt:<br />
linuxtrafficgather.pl<br />
<br />
== Monthly RAID checks ==<br />
<br />
Every month we check the health of and verfy the parity on all our RAID-based systems.<br />
To facilitate this, we've created a simple script to start the process:<br />
<br />
sh /root/verify.sh<br />
<br />
=== Adaptec controllers ===<br />
Here's some sample output:<br />
<pre><br />
mail /usr/local/www/scripts# sh /root/verify.sh<br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> container list /f<br />
Executing: container list /full=TRUE<br />
Num Total Oth Chunk Scsi Partition<br />
Creation System<br />
Label Type Size Ctr Size Usage B:ID:L Offset:Size State RO Lk Task Done% Ent<br />
Date Time Files<br />
----- ------ ------ --- ------ ------- ------ ------------- ------- -- -- ------- ------ ---<br />
------ -------- ------<br />
0 Mirror 33.9GB Open 0:01:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:32<br />
/dev/aacd0 mirror0 0:00:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:32<br />
<br />
1 Mirror 33.9GB Open 0:02:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:50<br />
/dev/aacd1 mirror1 0:03:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:50<br />
<br />
<br />
AAC0> disk list /f<br />
Executing: disk list /full=TRUE<br />
<br />
B:ID:L Device Type Removable media Vendor-ID Product-ID Rev Blocks Bytes/Bl<br />
ock Usage Shared Rate<br />
------ -------------- --------------- --------- ---------------- ----- --------- --------<br />
--- ---------------- ------ ----<br />
0:00:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:01:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:02:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:03:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
<br />
AAC0> disk show smart<br />
Executing: disk show smart<br />
<br />
Smart Method of Enable<br />
Capable Informational Exception Performance Error<br />
B:ID:L Device Exceptions(MRIE) Control Enabled Count<br />
------ ------- ---------------- --------- ----------- ------<br />
0:00:0 Y 6 Y N 0<br />
0:01:0 Y 6 Y N 0<br />
0:02:0 Y 6 Y N 0<br />
0:03:0 Y 6 Y N 0<br />
0:06:0 N<br />
<br />
AAC0> task list<br />
Executing: task list<br />
<br />
Controller Tasks<br />
<br />
TaskId Function Done% Container State Specific1 Specific2<br />
------ -------- ------- --------- ----- --------- ---------<br />
<br />
No tasks currently running on controller<br />
<br />
AAC0> dia sh hi<br />
Executing: diagnostic show history<br />
No switches specified, defaulting to "/current".<br />
<br />
<br />
<br />
*** HISTORY BUFFER FROM CURRENT CONTROLLER RUN ***<br />
<br />
[00]: GetDiskLogEntry: container - 1, entry return 0<br />
[01]: Container 1 started SCRUB task<br />
[02]: Starting Mirror:1 scrub<br />
[03]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[04]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[05]: UpdateDiskLogIndex - Set - container 0, index 1<br />
[06]: GetDiskLogEntry: container - 0, entry return 1<br />
[07]: Container 0 started SCRUB task<br />
[08]: Starting Mirror:0 scrub<br />
[09]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[10]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[11]: Mirror Scrub Container:1 ErrorsFound:0<br />
[12]: Clear disk log: sector - 80, driveno 2<br />
[13]: Clear disk log: sector - 80, driveno 3<br />
[14]: Container 1 completed SCRUB task:<br />
[15]: Mirror Scrub Container:0 ErrorsFound:0<br />
[16]: Clear disk log: sector - 81, driveno 1<br />
[17]: Clear disk log: sector - 81, driveno 0<br />
[18]: Container 0 completed SCRUB task:<br />
[19]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[20]: GetDiskLogEntry: container - 0, entry return 0<br />
[21]: Container 0 started SCRUB task<br />
[22]: Starting Mirror:0 scrub<br />
[23]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[24]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[25]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[26]: GetDiskLogEntry: container - 1, entry return 1<br />
[27]: Container 1 started SCRUB task<br />
[28]: Starting Mirror:1 scrub<br />
[29]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[30]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[31]: Mirror Scrub Container:1 ErrorsFound:0<br />
[32]: Clear disk log: sector - 81, driveno 2<br />
[33]: Clear disk log: sector - 81, driveno 3<br />
[34]: Container 1 completed SCRUB task:<br />
[35]: Mirror Scrub Container:0 ErrorsFound:0<br />
[36]: Clear disk log: sector - 80, driveno 1<br />
[37]: Clear disk log: sector - 80, driveno 0<br />
[38]: Container 0 completed SCRUB task:<br />
[39]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[40]: GetDiskLogEntry: container - 0, entry return 0<br />
[41]: Container 0 started SCRUB task<br />
[42]: Starting Mirror:0 scrub<br />
[43]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[44]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[45]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[46]: GetDiskLogEntry: container - 1, entry return 1<br />
[47]: Container 1 started SCRUB task<br />
[48]: Starting Mirror:1 scrub<br />
[49]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[50]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[51]: Mirror Scrub Container:1 ErrorsFound:0<br />
[52]: Clear disk log: sector - 81, driveno 2<br />
[53]: Clear disk log: sector - 81, driveno 3<br />
[54]: Container 1 completed SCRUB task:<br />
[55]: Mirror Scrub Container:0 ErrorsFound:0<br />
[56]: Clear disk log: sector - 80, driveno 1<br />
[57]: Clear disk log: sector - 80, driveno 0<br />
[58]: Container 0 completed SCRUB task:<br />
[59]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[60]: GetDiskLogEntry: container - 0, entry return 0<br />
[61]: Container 0 started SCRUB task<br />
[62]: Starting Mirror:0 scrub<br />
[63]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[64]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[65]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[66]: GetDiskLogEntry: container - 1, entry return 1<br />
[67]: Container 1 started SCRUB task<br />
[68]: Starting Mirror:1 scrub<br />
[69]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[70]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[71]: Mirror Scrub Container:1 ErrorsFound:0<br />
[72]: Clear disk log: sector - 81, driveno 2<br />
[73]: Clear disk log: sector - 81, driveno 3<br />
[74]: Container 1 completed SCRUB task:<br />
[75]: Mirror Scrub Container:0 ErrorsFound:0<br />
[76]: Clear disk log: sector - 80, driveno 1<br />
[77]: Clear disk log: sector - 80, driveno 0<br />
[78]: Container 0 completed SCRUB task:<br />
[79]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[80]: GetDiskLogEntry: container - 0, entry return 0<br />
[81]: Container 0 started SCRUB task<br />
[82]: Starting Mirror:0 scrub<br />
[83]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[84]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[85]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[86]: GetDiskLogEntry: container - 1, entry return 1<br />
[87]: Container 1 started SCRUB task<br />
[88]: Starting Mirror:1 scrub<br />
[89]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[90]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[91]: Mirror Scrub Container:1 ErrorsFound:0<br />
[92]: Clear disk log: sector - 81, driveno 2<br />
[93]: Clear disk log: sector - 81, driveno 3<br />
[94]: Container 1 completed SCRUB task:<br />
[95]: Mirror Scrub Container:0 ErrorsFound:0<br />
[96]: Clear disk log: sector - 80, driveno 1<br />
[97]: Clear disk log: sector - 80, driveno 0<br />
[98]: Container 0 completed SCRUB task:<br />
[99]:<br />
<br />
========================<br />
History Output Complete.<br />
<br />
AAC0><br />
AAC0> exit<br />
Executing: exit<br />
<br />
press enter when ready to run verify <INS><br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> contai scr 0<br />
Executing: container scrub 0<br />
<br />
AAC0> contai scr 1<br />
Executing: container scrub 1<br />
<br />
AAC0> exit<br />
Executing: exit<br />
<br />
when done run: <br />
<br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
<br />
<br />
Nov 1 10:32:46 mail /kernel: aac0: **Monitor** Container 0 started SCRUB task<br />
Nov 1 10:32:47 mail /kernel: aac0: **Monitor** Container 1 started SCRUB task<br />
<br />
</pre><br />
<br />
Here's an analysis of what we're seeing and what we're looking for:<br />
<br />
<pre><br />
AAC0> container list /f<br />
Executing: container list /full=TRUE<br />
Num Total Oth Chunk Scsi Partition<br />
Creation System<br />
Label Type Size Ctr Size Usage B:ID:L Offset:Size State RO Lk Task Done% Ent<br />
Date Time Files<br />
----- ------ ------ --- ------ ------- ------ ------------- ------- -- -- ------- ------ ---<br />
------ -------- ------<br />
0 Mirror 33.9GB Open 0:01:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:32<br />
/dev/aacd0 mirror0 0:00:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:32<br />
<br />
1 Mirror 33.9GB Open 0:02:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:50<br />
/dev/aacd1 mirror1 0:03:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:50<br />
</pre><br />
<br />
This is showing you the health of the arrays. You're looking for ''Normal'' under the State column, and the absence of a ! in the sector size - sometimes, you'll see this:<br />
64.0KB!33.9GB <br />
<br />
That indicates a problem.<br />
<br />
<pre><br />
AAC0> disk show smart<br />
Executing: disk show smart<br />
<br />
Smart Method of Enable<br />
Capable Informational Exception Performance Error<br />
B:ID:L Device Exceptions(MRIE) Control Enabled Count<br />
------ ------- ---------------- --------- ----------- ------<br />
0:00:0 Y 6 Y N 0<br />
0:01:0 Y 6 Y N 0<br />
0:02:0 Y 6 Y N 0<br />
0:03:0 Y 6 Y N 0<br />
0:06:0 N<br />
</pre><br />
<br />
This shows you a SMART report output. Looking for values in the Error Count column.<br />
<pre><br />
AAC0> task list<br />
Executing: task list<br />
<br />
Controller Tasks<br />
<br />
TaskId Function Done% Container State Specific1 Specific2<br />
------ -------- ------- --------- ----- --------- ---------<br />
<br />
No tasks currently running on controller<br />
</pre><br />
<br />
Look for absence of tasks running- a bad thing would be to see a rebuild or verify running when you didn't initiate it.<br />
<br />
With the history output, you're looking for any anomalies or events since the last time a verify was run. If you see a drive with lots of problems, you may want to take backups before allowing the verify to run since it could replicate errors onto the good drive.<br />
<br />
After you see the history output, it will prompt you to press enter to run the verify. If you're happy with all the output you're seeing- mirror is healthy, history looks good, it's safe to proceed. Otherwise ^C to exit. After hitting enter it will start the verify and start to tail the messages log file (so you can easily see when the verify is complete). Here's what that'll look like:<br />
<br />
<pre><br />
Nov 1 14:38:08 mail /kernel: aac0: **Monitor** Container 1 completed SCRUB task:<br />
Nov 1 14:46:45 mail /kernel: aac0: **Monitor** Container 0 completed SCRUB task:<br />
</pre><br />
<br />
So, putting it all together, after hitting enter to start the verify, you'll see:<br />
<br />
<pre><br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> contai scr 0<br />
Executing: container scrub 0<br />
<br />
AAC0> contai scr 1<br />
Executing: container scrub 1<br />
<br />
AAC0> exit<br />
Executing: exit<br />
<br />
when done run: <br />
<br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
<br />
<br />
Nov 1 10:32:46 mail /kernel: aac0: **Monitor** Container 0 started SCRUB task<br />
Nov 1 10:32:47 mail /kernel: aac0: **Monitor** Container 1 started SCRUB task<br />
<br />
</pre><br />
<br />
When the scrub(s) (verify) are complete - if the server has multiple logical drives, it will run both in parallel - you should exit the tail of the log file (^C) and run:<br />
<br />
<pre><br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
</pre><br />
<br />
Which will show you the diagnostic history, you're looking for the results of the most recent scrub:<br />
<br />
<pre><br />
[100]: Mirror Scrub Container:1 ErrorsFound:0<br />
[101]: Clear disk log: sector - 81, driveno 2<br />
[102]: Clear disk log: sector - 81, driveno 3<br />
[103]: Container 1 completed SCRUB task:<br />
[104]: Mirror Scrub Container:0 ErrorsFound:0<br />
[105]: Clear disk log: sector - 80, driveno 1<br />
[106]: Clear disk log: sector - 80, driveno 0<br />
[107]: Container 0 completed SCRUB task:<br />
</pre><br />
<br />
^C to exit the RAID CLI.<br />
<br />
If you see:<br />
<br />
[104]: Mirror Scrub Container:0 ErrorsFound:5<br />
<br />
You'll want to rerun the verify on that drive till it shows 0, or perhaps replace the drive- you should be able to see from the output which drive had the problem.<br />
<br />
Depending on the size and how busy the drive is, the verify can take anywhere from an hour to the better part of a day.<br />
<br />
You will notice that the diagnostic history is not shown on our modern adaptec cards (i.e. any adaptec card not in a Dell 2450). The reason for this is the history is never cleared, so there's simply too much data to show and it just crashes the CLI. So, don't bother trying to see it...which does make it hard to see if there are problems going on, so you just need to watch the scrub to see it goes to 100%. You will also notice that on some servers there's no tail of messages. Again, this is cause no data is shown there about the completion of the scrub. The thing to do here is to go into the CLI and continue to <tt>show tasks</tt> to monitor scrub progress.<br />
<br />
See [[RAIC_CLI#Adaptec|Adaptec RAID CLI Reference]] for more details on how to use the CLI<br />
<br />
=== DELL (LSI-based) SAS controllers ===<br />
<br />
Here's what the output looks like when running verify.sh on a LSI-based card:<br />
<br />
<pre><br />
jail2 /mnt/data2# sh /root/verify.sh<br />
<br />
Adapter #0<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 0<br />
Device Id: 0<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018396142<br />
SAS Address(1): 0x0<br />
Connected Port Number: 0(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DESN<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 1<br />
Device Id: 1<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018395db2<br />
SAS Address(1): 0x0<br />
Connected Port Number: 1(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DERV<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 2<br />
Device Id: 2<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50006eece89<br />
SAS Address(1): 0x0<br />
Connected Port Number: 2(path0)<br />
Inquiry Data: SEAGATE ST3300555SS T2113LM4BFBZ<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 3<br />
Device Id: 3<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50006eee035<br />
SAS Address(1): 0x0<br />
Connected Port Number: 3(path0)<br />
Inquiry Data: SEAGATE ST3300555SS T2113LM4BGF7<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 4<br />
Device Id: 4<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50004bd7ea5<br />
SAS Address(1): 0x0<br />
Connected Port Number: 4(path0)<br />
Inquiry Data: SEAGATE ST3300656SS HS093QP0G8SW<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 5<br />
Device Id: 5<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e01f1c4112<br />
SAS Address(1): 0x0<br />
Connected Port Number: 5(path0)<br />
Inquiry Data: FUJITSU MBA3300RC D306BJ15P9201W06<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 0 (Target Id: 0)<br />
Name:<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:139392MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 1 (Target Id: 1)<br />
Name:MIRROR1<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 2 (Target Id: 2)<br />
Name:MIRROR2<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
Battery FRU : N/A<br />
Battery Warning : Enabled<br />
Memory Correctable Errors : 0<br />
Memory Uncorrectable Errors : 0<br />
BBU : Present<br />
BBU : Yes<br />
Cache When BBU Bad : Disabled<br />
press enter when ready to run verify<br />
<br />
</pre><br />
<br />
Before pressing enter, here's what we're looking for:<br />
<br />
<pre><br />
Enclosure Device ID: 32<br />
Slot Number: 0<br />
Device Id: 0<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018396142<br />
SAS Address(1): 0x0<br />
Connected Port Number: 0(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DESN<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
</pre><br />
<br />
This is the output shown for each physical drive in the system. We're looking to confirm it's Firmware state is ''Online'', and Media Error Count, Other Error Count, and Predictive Failure Count are all zero (or near zero).<br />
<br />
<pre><br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 1 (Target Id: 1)<br />
Name:MIRROR1<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
</pre><br />
<br />
This is the output for each logical drive. We're looking for State ''Optimal''. Also confirm <tt>Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU</tt><br />
<br />
<pre><br />
Exit Code: 0x00<br />
Battery FRU : N/A<br />
Battery Warning : Enabled<br />
Memory Correctable Errors : 0<br />
Memory Uncorrectable Errors : 0<br />
BBU : Present<br />
BBU : Yes<br />
Cache When BBU Bad : Disabled<br />
</pre><br />
<br />
Confirm that the battery is present and error-free.<br />
<br />
If all that checks out, you're ready to proceed with the verify. After pressing enter, the verify is started and here's what you see:<br />
<pre><br />
Start Check Consistency on Virtual Drive 0 (target id: 0) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Start Check Consistency on Virtual Drive 1 (target id: 1) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Start Check Consistency on Virtual Drive 2 (target id: 2) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Check Consistency<br />
<br />
Progress of Virtual Drives...<br />
<br />
Virtual Drive # Percent Complete Time Elps<br />
0 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:03<br />
1 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:02<br />
2 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:01<br />
<br />
Press <ESC> key to quit...<br />
</pre><br />
<br />
The progress for each drive is displayed until all drives have completed the verify. We just want to make sure that each drive goes to completion. No followup is needed...though there probably is a log or history where we can get more info.<br />
<br />
You will notice that jail7 does not run a verify- that's on purpose. The last time we tried this it crashed the system. So, this must be run from the BIOS (take the system offline for a couple hours).<br />
<br />
See [[RAIC_CLI#LSI|LSI RAID CLI Reference]] for more details on how to use the CLI<br />
<br />
=== LSI-based controllers (megaraid) ===<br />
<br />
There is a CLI for this however it's easier to do this with a curses GUI app: megaraid<br />
<br />
Currently only on these servers: virt15, virt16, and firewall2<br />
<br />
To run:<br />
<br />
# cd /usr/local/sbin/; megamgr<br />
<br />
Main menu:<br />
<pre>²ÚÄÄManagement MenuÄÄ¿²<br />
²³ Configure ³²<br />
²³ Initialize ³²<br />
²³ Objects ³²<br />
²³ Rebuild ³²<br />
²³ Check Consistency ³²<br />
²³ Advanced Menu ³²<br />
²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²</pre><br />
<br />
Before you check consistency, make sure the arrays are healthy.<br />
<br />
Objects -> Physical Drive<br />
<br />
Then look to make sure they're all <tt>ONLIN</tt><br />
<pre><br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄObjects - PHYSICAL DRIVE SELECTION MENUÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²<br />
²²²²²²²²³ ³²²²²²²<br />
²²²²²²²²³ Channel-1 ³²²²²²²<br />
²ÚÄÄMana³ ID ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»x ³²²²²²²<br />
²³ Confi³ 0º* ONLIN A01-01º ³²²²²²²<br />
²³ Initi³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²³ Objec³ 1º* ONLIN A01-02º ³²²²²²²<br />
²³ Rebui³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²³ Check³ 2º* ONLIN A02-01º ³²²²²²²<br />
²³ Advan³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²ÀÄÄÄÄÄij 3º* ONLIN A02-02º ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 4º* ONLIN A03-01º ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 5º* ONLIN A03-02ºþ ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 6º* º ³²²²²²²<br />
²²²²²²²²³ ÈÍÍÍÍÍÍÍÍÍÍÍÍÍͼx ³²²²²²²<br />
²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²<br />
²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²²²<br />
²²²²²²²²³Ch-1 ID-5 DISK 140013MB SEAGATE ST3146707LC 0003 ³²²²²²²²²<br />
²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²<br />
</pre><br />
<br />
<br />
Once that's done, hit escape once then the back arror to move back to the Objects menu.<br />
So you select Objects -> Logical Drive -> Logical Drive 1 -> Check Consistency -> YES<br />
<br />
<pre><br />
<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÚÄLogical Drives(02)Ä¿²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ Logical Drive 1 ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²ÚÄÄÄÄObjectsÄÄij Logical Drive 2 ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²ÚÄÄManagemen³ Adapter ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Configure ³ Logical Drive ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Initialize³ Physical Drive ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Objects ³ Channel ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Rebuild ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Check Consistency ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Advanced Menu ³²²²²²²²ÚÄÄÄÄLogical Drive 1ÄÄÄÄÄ¿²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²³ Initialize ÚÄCheck Consistency-1 ?Ä¿²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ Check Consiste³ YES ³²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ View/Update Pa³ NO ³²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³Select YES Or NO³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
</pre><br />
<br />
Then watch the progress. When done, escape back to Logical Drive then repeat for Logical Drive 2. If you ^C or accidentally escape out, you can come back in running the same commands and watch the progress again (it won't restart).<br />
<br />
You can exit megamgr by escaping out or ^C<br />
<br />
=== 3ware ===<br />
<br />
We are using 3ware controllers on backup1 & backup2. Running the verify script will give you different output based on the type of controller:<br />
<br />
<pre><br />
backup2 /d2# sh /root/verify.sh<br />
Controller: c0<br />
-------------<br />
Driver: 1.50.01.002<br />
Model: 7500-8<br />
FW: FE7X 1.05.00.068<br />
BIOS: BE7X 1.08.00.048<br />
Monitor: ME7X 1.01.00.040<br />
Serial #: F11605A3180172<br />
PCB: Rev3<br />
PCHIP: 1.30-33<br />
ACHIP: 3.20<br />
<br />
<br />
# of units: 3<br />
Unit 0: JBOD 186.31 GB ( 390721968 blocks): OK<br />
Unit 1: RAID 5 465.77 GB ( 976790016 blocks): DEGRADED<br />
Unit 5: RAID 5 698.65 GB ( 1465185024 blocks): DEGRADED<br />
<br />
# of ports: 8<br />
Port 0: WDC WD2000JB-00KFA0 WD-WCAMT1451690 186.31 GB (390721968 blocks): OK(unit 0)<br />
Port 1: WDC WD2500JB-00GVC0 WD-WCAL78219488 232.88 GB (488397168 blocks): OK(unit 1)<br />
Port 2: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT)<br />
Port 3: WDC WD2500JB-00GVC0 WD-WMAL73882417 232.88 GB (488397168 blocks): OK(unit 1)<br />
Port 4: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT)<br />
Port 5: WDC WD2500JB-00GVA0 WD-WMAL71338097 232.88 GB (488397168 blocks): OK(unit 5)<br />
Port 6: WDC WD2500JB-32EVA0 WD-WMAEH1301595 232.88 GB (488397168 blocks): OK(unit 5)<br />
Port 7: WDC WD2500JB-00GVC0 WD-WCAL78165566 232.88 GB (488397168 blocks): OK(unit 5)<br />
Controller: c1<br />
-------------<br />
Driver: 1.50.01.002<br />
Model: 7500-8<br />
FW: FE7X 1.05.00.068<br />
BIOS: BE7X 1.08.00.048<br />
Monitor: ME7X 1.01.00.040<br />
Serial #: F11605A3180167<br />
PCB: Rev3<br />
PCHIP: 1.30-33<br />
ACHIP: 3.20<br />
<br />
<br />
# of units: 2<br />
Unit 0: RAID 5 698.65 GB ( 1465185024 blocks): OK<br />
Unit 4: RAID 5 698.65 GB ( 1465185024 blocks): OK<br />
<br />
# of ports: 8<br />
Port 0: WDC WD2500JB-00GVA0 WD-WMAL71301258 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 1: WDC WD2500JB-00GVA0 WD-WMAL71322705 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 2: WDC WD2500JB-00GVA0 WD-WMAL71945050 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 3: WDC WD2500JB-00GVA0 WD-WMAL71316201 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 4: WDC WD2500JB-00GVC0 WD-WCAL78323749 232.88 GB (488397168 blocks): OK(unit 4)<br />
Port 5: WDC WD3200AAJB-00J3A0 WD-WCAV2V689068 298.09 GB (625142448 blocks): OK(unit 4)<br />
Port 6: WDC WD2500JB-00GVC0 WD-WCAL78234420 232.88 GB (488397168 blocks): OK(unit 4)<br />
Port 7: WDC WD2500JB-00GVC0 WD-WCAL78592213 232.88 GB (488397168 blocks): OK(unit 4)<br />
backup2 /d2#<br />
</pre><br />
<br />
On backup2 look for all ok, no verify.<br />
<br />
<pre><br />
[root@backup3 ~]# sh /root/verify.sh<br />
/c2 Driver Version = 1.26.02.002<br />
/c2 Model = 8006-2LP<br />
/c2 Available Memory = 512KB<br />
/c2 Firmware Version = FE8S 1.05.00.068<br />
/c2 Bios Version = BE7X 1.08.00.048<br />
/c2 Boot Loader Version = ME7X 1.01.00.040<br />
/c2 Serial Number = L018501C6481395<br />
/c2 PCB Version = Rev5<br />
/c2 PCHIP Version = 1.30-66<br />
/c2 ACHIP Version = 3.20<br />
/c2 Total Optimal Units = 1<br />
/c2 Not Optimal Units = 0<br />
<br />
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u1 RAID-1 OK - - - 931.512 ON -<br />
<br />
Port Status Unit Size Blocks Serial<br />
---------------------------------------------------------------<br />
p0 OK u1 931.51 GB 1953525168 WD-WMAW31148820<br />
p1 OK u1 931.51 GB 1953525168 WD-WCATR0277515<br />
<br />
<br />
Ctl Date Severity Alarm Message<br />
------------------------------------------------------------------------------<br />
<br />
Sending start verify message to /c2/u1 ... Done.<br />
<br />
<br />
when done run:<br />
tw_cli /c2 show alarms<br />
<br />
[root@backup3 ~]#<br />
</pre><br />
<br />
Automatically starts the verify, just run <tt>tw_cli /c2 show alarms</tt> as instructed to see the results of the verify.<br />
<br />
<pre><br />
[root@backup1 /data/deprecated]# sh /root/verify.sh<br />
/c0 Driver Version = 2.26.02.010<br />
/c0 Model = 9650SE-8LPML<br />
/c0 Available Memory = 224MB<br />
/c0 Firmware Version = FE9X 4.06.00.004<br />
/c0 Bios Version = BE9X 4.05.00.015<br />
/c0 Boot Loader Version = BL9X 3.08.00.001<br />
/c0 Serial Number = L326025A8270177<br />
/c0 PCB Version = Rev 032<br />
/c0 PCHIP Version = 2.00<br />
/c0 ACHIP Version = 1.90<br />
/c0 Number of Ports = 8<br />
/c0 Number of Drives = 6<br />
/c0 Number of Units = 1<br />
/c0 Total Optimal Units = 1<br />
/c0 Not Optimal Units = 0<br />
/c0 JBOD Export Policy = off<br />
/c0 Disk Spinup Policy = 1<br />
/c0 Spinup Stagger Time Policy (sec) = 1<br />
/c0 Auto-Carving Policy = off<br />
/c0 Auto-Carving Size = 2048 GB<br />
/c0 Auto-Rebuild Policy = on<br />
/c0 Controller Bus Type = PCIe<br />
/c0 Controller Bus Width = 1 lane<br />
/c0 Controller Bus Speed = 2.5 Gbps/lane<br />
<br />
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u0 RAID-5 OK - - 64K 4656.56 ON ON<br />
<br />
Port Status Unit Size Blocks Serial<br />
---------------------------------------------------------------<br />
p0 OK u0 931.51 GB 1953525168 9QJ1Y017<br />
p1 DEVICE-ERROR u0 931.51 GB 1953525168 9QJ1ZN07<br />
p2 OK u0 931.51 GB 1953525168 9QJ2XK1R<br />
p3 OK u0 931.51 GB 1953525168 9QJ2010B<br />
p4 OK u0 1.36 TB 2930277168 6XW0L36T<br />
p5 OK u0 931.51 GB 1953525168 WD-WMATV2444836<br />
p6 NOT-PRESENT - - - -<br />
p7 NOT-PRESENT - - - -<br />
<br />
<br />
Ctl Date Severity Alarm Message<br />
------------------------------------------------------------------------------<br />
c0 [Sat May 12 11:27:15 2012] WARNING Sector repair completed: port=0, LBA=0x6AE571C<br />
c0 [Sat May 12 19:16:21 2012] WARNING Sector repair completed: port=1, LBA=0x40E62A23<br />
c0 [Sat May 12 21:40:56 2012] INFO Verify completed: unit=0<br />
c0 [Mon May 14 00:53:53 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFA7<br />
c0 [Mon May 14 00:58:21 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAA<br />
c0 [Mon May 14 04:35:13 2012] WARNING Sector repair completed: port=0, LBA=0x8FEF2CF<br />
c0 [Mon May 14 04:38:22 2012] WARNING Sector repair completed: port=0, LBA=0x8FEF2D1<br />
c0 [Tue May 15 22:53:46 2012] WARNING Sector repair completed: port=0, LBA=0x13C2622<br />
c0 [Wed May 16 00:39:31 2012] WARNING Sector repair completed: port=0, LBA=0x365A67F<br />
c0 [Wed May 16 00:39:37 2012] WARNING Sector repair completed: port=0, LBA=0x365A685<br />
c0 [Wed May 16 00:47:18 2012] WARNING Sector repair completed: port=0, LBA=0x365A687<br />
c0 [Sat May 19 00:01:44 2012] INFO Verify started: unit=0<br />
c0 [Sat May 19 04:46:20 2012] WARNING Sector repair completed: port=0, LBA=0x365A68E<br />
c0 [Sat May 19 13:37:06 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAC<br />
c0 [Sat May 19 13:37:28 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAE<br />
c0 [Sat May 19 13:37:47 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFB1<br />
c0 [Sat May 19 13:38:00 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFB3<br />
c0 [Sat May 19 21:47:45 2012] INFO Verify completed: unit=0<br />
c0 [Wed May 23 12:21:41 2012] INFO Cache synchronization completed: unit=0<br />
c0 [Fri May 25 00:08:19 2012] WARNING Sector repair completed: port=0, LBA=0x12DA76C<br />
c0 [Fri May 25 00:08:34 2012] WARNING Sector repair completed: port=0, LBA=0x12E4901<br />
c0 [Fri May 25 00:09:33 2012] WARNING Sector repair completed: port=0, LBA=0x12DA773<br />
c0 [Fri May 25 00:39:12 2012] WARNING Sector repair completed: port=0, LBA=0x42C597B<br />
c0 [Sat May 26 00:01:45 2012] INFO Verify started: unit=0<br />
c0 [Sat May 26 00:42:05 2012] WARNING Sector repair completed: port=1, LBA=0x323C1AC<br />
c0 [Sat May 26 00:51:43 2012] WARNING Sector repair completed: port=1, LBA=0x323C1AE<br />
c0 [Sat May 26 01:54:21 2012] WARNING Sector repair completed: port=1, LBA=0x2F0D302<br />
c0 [Sat May 26 02:06:38 2012] WARNING Sector repair completed: port=0, LBA=0x12DA777<br />
c0 [Sat May 26 02:07:21 2012] WARNING Sector repair completed: port=0, LBA=0x12E48FE<br />
c0 [Sat May 26 04:20:00 2012] WARNING Sector repair completed: port=1, LBA=0x2F0D306<br />
c0 [Sat May 26 04:32:58 2012] WARNING Sector repair completed: port=1, LBA=0x323C1B1<br />
c0 [Sat May 26 04:33:21 2012] WARNING Sector repair completed: port=1, LBA=0x323C1B3<br />
c0 [Sat May 26 04:33:44 2012] WARNING Sector repair completed: port=1, LBA=0x323C1BA<br />
c0 [Sat May 26 05:24:07 2012] WARNING Sector repair completed: port=1, LBA=0x3F83862<br />
c0 [Sat May 26 05:25:09 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 06:08:13 2012] WARNING Sector repair completed: port=0, LBA=0x4CDC6A2<br />
c0 [Sat May 26 09:49:35 2012] WARNING Sector repair completed: port=1, LBA=0x6CACD4A<br />
c0 [Sat May 26 18:10:44 2012] WARNING Sector repair completed: port=1, LBA=0x18F425EA<br />
c0 [Sat May 26 19:45:40 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:22:52 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:15 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:22 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:35 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:41 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:49 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:57 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:24:02 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:54:41 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 22:00:30 2012] INFO Verify completed: unit=0<br />
c0 [Sat Jun 2 00:01:43 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 2 00:30:17 2012] WARNING Sector repair completed: port=0, LBA=0x2B911E4<br />
c0 [Sat Jun 2 00:50:57 2012] WARNING Sector repair completed: port=0, LBA=0x5A807CA6<br />
c0 [Sat Jun 2 04:13:13 2012] WARNING Sector repair completed: port=0, LBA=0x2D18291<br />
c0 [Sat Jun 2 04:13:35 2012] WARNING Sector repair completed: port=0, LBA=0x2D1829F<br />
c0 [Sat Jun 2 21:48:02 2012] INFO Verify completed: unit=0<br />
c0 [Mon Jun 4 04:40:34 2012] WARNING Sector repair completed: port=1, LBA=0x4AF8098F<br />
c0 [Tue Jun 5 00:28:19 2012] WARNING Sector repair completed: port=1, LBA=0x263C5CD<br />
c0 [Tue Jun 5 00:33:06 2012] WARNING Sector repair completed: port=1, LBA=0x263C5CF<br />
c0 [Thu Jun 7 00:34:27 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B5F<br />
c0 [Thu Jun 7 00:38:50 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B61<br />
c0 [Fri Jun 8 00:07:13 2012] WARNING Sector repair completed: port=0, LBA=0xC131F6B<br />
c0 [Sat Jun 9 00:01:41 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 9 00:29:11 2012] WARNING Sector repair completed: port=0, LBA=0x6C7614D<br />
c0 [Sat Jun 9 00:38:25 2012] WARNING Sector repair completed: port=0, LBA=0x6C76152<br />
c0 [Sat Jun 9 04:02:30 2012] WARNING Sector repair completed: port=1, LBA=0x263C5D1<br />
c0 [Sat Jun 9 04:02:52 2012] WARNING Sector repair completed: port=1, LBA=0x263C5D3<br />
c0 [Sat Jun 9 04:07:32 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E12<br />
c0 [Sat Jun 9 04:07:57 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E15<br />
c0 [Sat Jun 9 04:08:16 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E17<br />
c0 [Sat Jun 9 04:08:45 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E19<br />
c0 [Sat Jun 9 04:15:04 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B64<br />
c0 [Sat Jun 9 04:15:26 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B66<br />
c0 [Sat Jun 9 04:15:45 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B68<br />
c0 [Sat Jun 9 04:15:59 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B6C<br />
c0 [Sat Jun 9 04:16:13 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B6E<br />
c0 [Sat Jun 9 21:48:52 2012] INFO Verify completed: unit=0<br />
c0 [Thu Jun 14 00:40:10 2012] WARNING Sector repair completed: port=0, LBA=0x334F14B<br />
c0 [Sat Jun 16 00:01:38 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 16 21:16:19 2012] INFO Verify completed: unit=0<br />
c0 [Tue Jun 19 02:03:43 2012] WARNING Sector repair completed: port=1, LBA=0xFE41EAD<br />
c0 [Wed Jun 20 02:30:02 2012] WARNING Sector repair completed: port=1, LBA=0xD99145C<br />
c0 [Sat Jun 23 00:01:36 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 23 04:27:04 2012] WARNING Sector repair completed: port=1, LBA=0x2FAD311<br />
c0 [Sat Jun 23 06:52:38 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC8D<br />
c0 [Sat Jun 23 06:53:03 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC91<br />
c0 [Sat Jun 23 06:53:21 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC94<br />
c0 [Sat Jun 23 17:00:22 2012] WARNING Sector repair completed: port=1, LBA=0xF9AC7C9<br />
c0 [Sat Jun 23 21:15:19 2012] INFO Verify completed: unit=0<br />
c0 [Sat Jun 30 00:01:34 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 30 05:24:13 2012] WARNING Sector repair completed: port=0, LBA=0x3FAA9E7<br />
c0 [Sat Jun 30 14:49:39 2012] WARNING Sector repair completed: port=1, LBA=0x869931C<br />
c0 [Sat Jun 30 21:31:05 2012] INFO Verify completed: unit=0<br />
c0 [Tue Jul 3 03:40:25 2012] WARNING Sector repair completed: port=1, LBA=0xD36C7F7<br />
c0 [Fri Jul 6 02:50:18 2012] WARNING Sector repair completed: port=1, LBA=0x3562470<br />
c0 [Fri Jul 6 22:18:26 2012] WARNING Sector repair completed: port=1, LBA=0x3563173<br />
c0 [Sat Jul 7 00:01:31 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 7 00:50:16 2012] WARNING Sector repair completed: port=0, LBA=0x76EE88<br />
c0 [Sat Jul 7 00:50:39 2012] WARNING Sector repair completed: port=0, LBA=0x76EE8F<br />
c0 [Sat Jul 7 21:39:36 2012] INFO Verify completed: unit=0<br />
c0 [Sun Jul 8 02:51:05 2012] WARNING Sector repair completed: port=0, LBA=0x67759D<br />
c0 [Sun Jul 8 02:53:55 2012] WARNING Sector repair completed: port=0, LBA=0x67759B<br />
c0 [Tue Jul 10 16:17:21 2012] WARNING Sector repair completed: port=0, LBA=0x15C8C695<br />
c0 [Wed Jul 11 22:51:22 2012] WARNING Sector repair completed: port=1, LBA=0x355BBD0<br />
c0 [Sat Jul 14 00:01:28 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 14 01:33:40 2012] WARNING Sector repair completed: port=1, LBA=0x1333BCF4<br />
c0 [Sat Jul 14 03:36:23 2012] WARNING Sector repair completed: port=1, LBA=0x2174773<br />
c0 [Sat Jul 14 11:26:44 2012] WARNING Sector repair completed: port=0, LBA=0x7429AB7<br />
c0 [Sat Jul 14 16:53:50 2012] WARNING Sector repair completed: port=1, LBA=0xA17EB3F<br />
c0 [Sat Jul 14 21:19:25 2012] INFO Verify completed: unit=0<br />
c0 [Wed Jul 18 05:08:47 2012] WARNING Sector repair completed: port=1, LBA=0x17D62EDC<br />
c0 [Wed Jul 18 05:14:15 2012] WARNING Sector repair completed: port=1, LBA=0x17D62EE1<br />
c0 [Thu Jul 19 03:24:59 2012] WARNING Sector repair completed: port=0, LBA=0x7733C3D<br />
c0 [Thu Jul 19 03:25:20 2012] WARNING Sector repair completed: port=0, LBA=0x773CEA5<br />
c0 [Thu Jul 19 03:28:16 2012] WARNING Sector repair completed: port=0, LBA=0x7733C42<br />
c0 [Thu Jul 19 03:28:41 2012] WARNING Sector repair completed: port=0, LBA=0x773CEAF<br />
c0 [Sat Jul 21 00:01:26 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 21 03:07:30 2012] WARNING Sector repair completed: port=1, LBA=0x1CC6936<br />
c0 [Sat Jul 21 03:07:52 2012] WARNING Sector repair completed: port=1, LBA=0x1CC6938<br />
c0 [Sat Jul 21 03:08:11 2012] WARNING Sector repair completed: port=1, LBA=0x1CC693A<br />
c0 [Sat Jul 21 16:43:56 2012] WARNING Sector repair completed: port=0, LBA=0xD04C914<br />
c0 [Sat Jul 21 16:45:31 2012] WARNING Sector repair completed: port=1, LBA=0xD456973<br />
c0 [Sat Jul 21 21:14:29 2012] INFO Verify completed: unit=0<br />
c0 [Wed Jul 25 03:37:25 2012] WARNING Sector repair completed: port=0, LBA=0x1F8E6C43<br />
c0 [Sat Jul 28 00:01:24 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 28 01:45:27 2012] WARNING Sector repair completed: port=0, LBA=0x11584AD<br />
c0 [Sat Jul 28 18:54:25 2012] WARNING Sector repair completed: port=1, LBA=0x447C3E6C<br />
c0 [Sat Jul 28 21:13:46 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 1 03:20:11 2012] WARNING Sector repair completed: port=0, LBA=0x805FEF<br />
c0 [Fri Aug 3 00:50:03 2012] WARNING Sector repair completed: port=0, LBA=0xCED0ACA<br />
c0 [Sat Aug 4 00:01:22 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 4 00:52:51 2012] WARNING Sector repair completed: port=0, LBA=0x805FF3<br />
c0 [Sat Aug 4 00:53:14 2012] WARNING Sector repair completed: port=0, LBA=0x805FF5<br />
c0 [Sat Aug 4 00:53:33 2012] WARNING Sector repair completed: port=0, LBA=0x805FF7<br />
c0 [Sat Aug 4 00:53:47 2012] WARNING Sector repair completed: port=0, LBA=0x805FF9<br />
c0 [Sat Aug 4 00:54:00 2012] WARNING Sector repair completed: port=0, LBA=0x805FFB<br />
c0 [Sat Aug 4 00:54:14 2012] WARNING Sector repair completed: port=0, LBA=0x805FFD<br />
c0 [Sat Aug 4 00:54:27 2012] WARNING Sector repair completed: port=0, LBA=0x805FFF<br />
c0 [Sat Aug 4 04:43:12 2012] WARNING Sector repair completed: port=1, LBA=0x16974289<br />
c0 [Sat Aug 4 04:58:17 2012] WARNING Sector repair completed: port=1, LBA=0x1697428E<br />
c0 [Sat Aug 4 20:54:53 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 8 03:21:55 2012] ERROR Drive timeout detected: port=1<br />
c0 [Wed Aug 8 15:31:44 2012] WARNING Sector repair completed: port=0, LBA=0x1A366CD3<br />
c0 [Sat Aug 11 00:01:21 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 11 20:40:51 2012] INFO Verify completed: unit=0<br />
c0 [Thu Aug 16 05:10:55 2012] WARNING Sector repair completed: port=0, LBA=0x1C22593<br />
c0 [Sat Aug 18 00:01:18 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 18 03:00:20 2012] WARNING Sector repair completed: port=0, LBA=0x1C225A5<br />
c0 [Sat Aug 18 03:43:00 2012] WARNING Sector repair completed: port=1, LBA=0x23EE91E<br />
c0 [Sat Aug 18 03:43:23 2012] WARNING Sector repair completed: port=1, LBA=0x23EE920<br />
c0 [Sat Aug 18 17:00:06 2012] WARNING Sector repair completed: port=1, LBA=0x137D066A<br />
c0 [Sat Aug 18 17:00:29 2012] WARNING Sector repair completed: port=1, LBA=0x137D066D<br />
c0 [Sat Aug 18 21:13:01 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 22 01:36:08 2012] WARNING Sector repair completed: port=0, LBA=0x2560A0F<br />
c0 [Wed Aug 22 01:37:42 2012] WARNING Sector repair completed: port=0, LBA=0x2560A13<br />
c0 [Fri Aug 24 04:01:36 2012] WARNING Sector repair completed: port=1, LBA=0x55C1A5DF<br />
c0 [Fri Aug 24 05:02:06 2012] WARNING Sector repair completed: port=1, LBA=0xCE3378A<br />
c0 [Sat Aug 25 00:01:17 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 25 00:31:06 2012] WARNING Sector repair completed: port=1, LBA=0x50F65D<br />
c0 [Sat Aug 25 00:39:52 2012] WARNING Sector repair completed: port=0, LBA=0x678FF4<br />
c0 [Sat Aug 25 03:43:15 2012] WARNING Sector repair completed: port=0, LBA=0x2560A15<br />
c0 [Sat Aug 25 03:43:39 2012] WARNING Sector repair completed: port=0, LBA=0x2560A19<br />
c0 [Sat Aug 25 03:43:58 2012] WARNING Sector repair completed: port=0, LBA=0x2560A1B<br />
c0 [Sat Aug 25 03:44:30 2012] WARNING Sector repair completed: port=0, LBA=0x2560A21<br />
c0 [Sat Aug 25 20:58:14 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 29 04:57:15 2012] WARNING Sector repair completed: port=1, LBA=0xF3957EB<br />
c0 [Sat Sep 1 00:01:15 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 1 03:21:52 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC86<br />
c0 [Sat Sep 1 03:22:15 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC88<br />
c0 [Sat Sep 1 03:22:34 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC8A<br />
c0 [Sat Sep 1 03:22:47 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC8C<br />
c0 [Sat Sep 1 17:17:22 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD1<br />
c0 [Sat Sep 1 17:17:45 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD3<br />
c0 [Sat Sep 1 17:18:04 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD5<br />
c0 [Sat Sep 1 21:36:56 2012] INFO Verify completed: unit=0<br />
c0 [Thu Sep 6 00:07:30 2012] WARNING Sector repair completed: port=0, LBA=0xDA3C64B<br />
c0 [Thu Sep 6 00:32:56 2012] WARNING Sector repair completed: port=1, LBA=0x6BBA816<br />
c0 [Sat Sep 8 00:01:13 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 8 00:09:56 2012] WARNING Sector repair completed: port=0, LBA=0xDEBC958<br />
c0 [Sat Sep 8 04:38:45 2012] WARNING Sector repair completed: port=0, LBA=0x38D254F<br />
c0 [Sat Sep 8 20:44:50 2012] INFO Verify completed: unit=0<br />
c0 [Mon Sep 10 01:26:34 2012] WARNING Sector repair completed: port=1, LBA=0xFFD8D5E<br />
c0 [Wed Sep 12 00:33:48 2012] WARNING Sector repair completed: port=1, LBA=0xE8DB928<br />
c0 [Wed Sep 12 00:36:33 2012] WARNING Sector repair completed: port=1, LBA=0x6D49411<br />
c0 [Fri Sep 14 01:59:39 2012] WARNING Sector repair completed: port=0, LBA=0x1467F1C<br />
c0 [Fri Sep 14 02:08:27 2012] WARNING Sector repair completed: port=1, LBA=0x14C8ABD<br />
c0 [Fri Sep 14 03:54:47 2012] WARNING Sector repair completed: port=0, LBA=0x1580C915<br />
c0 [Sat Sep 15 00:01:11 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 15 02:38:14 2012] WARNING Sector repair completed: port=0, LBA=0x1C178973<br />
c0 [Sat Sep 15 02:59:02 2012] WARNING Sector repair completed: port=0, LBA=0x1C178975<br />
c0 [Sat Sep 15 04:47:08 2012] WARNING Sector repair completed: port=0, LBA=0x3FA0356<br />
c0 [Sat Sep 15 04:47:31 2012] WARNING Sector repair completed: port=0, LBA=0x3FA0359<br />
c0 [Sat Sep 15 10:41:59 2012] WARNING Sector repair completed: port=0, LBA=0x6DFD1EC<br />
c0 [Sat Sep 15 13:25:23 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD100<br />
c0 [Sat Sep 15 13:25:31 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD104<br />
c0 [Sat Sep 15 13:25:54 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD106<br />
c0 [Sat Sep 15 17:10:50 2012] WARNING Sector repair completed: port=0, LBA=0x1C178977<br />
c0 [Sat Sep 15 20:59:57 2012] INFO Verify completed: unit=0<br />
c0 [Tue Sep 18 01:17:18 2012] WARNING Sector repair completed: port=1, LBA=0x803B05B<br />
c0 [Sat Sep 22 00:01:10 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 22 20:54:31 2012] INFO Verify completed: unit=0<br />
c0 [Tue Sep 25 01:56:47 2012] WARNING Sector repair completed: port=0, LBA=0x26E3909<br />
c0 [Sat Sep 29 00:01:08 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 29 02:04:14 2012] WARNING Sector repair completed: port=0, LBA=0x146AC03<br />
c0 [Sat Sep 29 10:58:39 2012] WARNING Sector repair completed: port=0, LBA=0x6D4EB0E<br />
c0 [Sat Sep 29 10:59:02 2012] WARNING Sector repair completed: port=0, LBA=0x6D4EB14<br />
c0 [Sat Sep 29 11:22:44 2012] WARNING Sector repair completed: port=0, LBA=0x6F79623<br />
c0 [Sat Sep 29 13:50:48 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D65E<br />
c0 [Sat Sep 29 13:51:11 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D661<br />
c0 [Sat Sep 29 13:51:30 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D663<br />
c0 [Sat Sep 29 20:57:34 2012] INFO Verify completed: unit=0<br />
c0 [Mon Oct 1 04:47:24 2012] WARNING Sector repair completed: port=0, LBA=0xC5BC6F2<br />
c0 [Tue Oct 2 02:00:27 2012] WARNING Sector repair completed: port=0, LBA=0x1547667<br />
c0 [Tue Oct 2 02:01:56 2012] WARNING Sector repair completed: port=0, LBA=0x154766F<br />
c0 [Tue Oct 2 05:02:31 2012] WARNING Sector repair completed: port=1, LBA=0xD67D054<br />
c0 [Tue Oct 2 05:04:14 2012] WARNING Sector repair completed: port=1, LBA=0xD67D056<br />
c0 [Wed Oct 3 01:22:12 2012] WARNING Sector repair completed: port=1, LBA=0x12AAF8CA<br />
c0 [Thu Oct 4 04:29:22 2012] WARNING Sector repair completed: port=0, LBA=0x13E6F992<br />
c0 [Thu Oct 4 05:10:51 2012] WARNING Sector repair completed: port=0, LBA=0x1C252A4<br />
c0 [Sat Oct 6 00:01:07 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 6 19:41:18 2012] WARNING Sector repair completed: port=1, LBA=0x5A5C3AE8<br />
c0 [Sat Oct 6 21:01:05 2012] INFO Verify completed: unit=0<br />
c0 [Mon Oct 8 00:32:06 2012] WARNING Sector repair completed: port=0, LBA=0x6C60D3E<br />
c0 [Tue Oct 9 03:51:03 2012] WARNING Sector repair completed: port=1, LBA=0x89B5EC9<br />
c0 [Thu Oct 11 04:21:17 2012] WARNING Sector repair completed: port=1, LBA=0x13F85833<br />
c0 [Sat Oct 13 00:01:05 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 13 05:12:40 2012] WARNING Sector repair completed: port=0, LBA=0x3FA5134<br />
c0 [Sat Oct 13 21:08:35 2012] INFO Verify completed: unit=0<br />
c0 [Tue Oct 16 03:53:50 2012] WARNING Sector repair completed: port=1, LBA=0x148AA1BD<br />
c0 [Thu Oct 18 03:20:30 2012] WARNING Sector repair completed: port=0, LBA=0x1C8DABCB<br />
c0 [Thu Oct 18 04:52:50 2012] WARNING Sector repair completed: port=0, LBA=0xE879057<br />
c0 [Sat Oct 20 00:01:04 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 20 02:19:25 2012] WARNING Sector repair completed: port=1, LBA=0x174B012<br />
c0 [Sat Oct 20 03:41:38 2012] WARNING Sector repair completed: port=0, LBA=0x256D93B<br />
c0 [Sat Oct 20 03:42:01 2012] WARNING Sector repair completed: port=0, LBA=0x256D93D<br />
c0 [Sat Oct 20 03:42:40 2012] WARNING Sector repair completed: port=0, LBA=0x256D940<br />
c0 [Sat Oct 20 03:42:59 2012] WARNING Sector repair completed: port=0, LBA=0x256D942<br />
c0 [Sat Oct 20 03:43:12 2012] WARNING Sector repair completed: port=0, LBA=0x256D944<br />
c0 [Sat Oct 20 03:43:26 2012] WARNING Sector repair completed: port=0, LBA=0x256D948<br />
c0 [Sat Oct 20 16:37:52 2012] WARNING Sector repair completed: port=0, LBA=0xE879060<br />
c0 [Sat Oct 20 16:38:15 2012] WARNING Sector repair completed: port=0, LBA=0xE879062<br />
c0 [Sat Oct 20 21:00:18 2012] INFO Verify completed: unit=0<br />
c0 [Sat Oct 20 23:49:01 2012] WARNING Sector repair completed: port=1, LBA=0x4473E908<br />
c0 [Sun Oct 21 03:42:26 2012] WARNING Sector repair completed: port=1, LBA=0x175BADD5<br />
c0 [Tue Oct 23 01:09:04 2012] WARNING Sector repair completed: port=1, LBA=0x6E524860<br />
c0 [Fri Oct 26 03:21:25 2012] WARNING Sector repair completed: port=0, LBA=0x802C61<br />
c0 [Fri Oct 26 04:22:21 2012] WARNING Sector repair completed: port=0, LBA=0x176353CD<br />
c0 [Sat Oct 27 00:01:03 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 27 00:49:35 2012] WARNING Sector repair completed: port=0, LBA=0x802C65<br />
c0 [Sat Oct 27 17:02:24 2012] WARNING Sector repair completed: port=1, LBA=0xC1FF26D<br />
c0 [Sat Oct 27 17:09:06 2012] WARNING Sector repair completed: port=0, LBA=0xDF621AD<br />
c0 [Sat Oct 27 21:30:57 2012] INFO Verify completed: unit=0<br />
c0 [Tue Oct 30 00:20:46 2012] WARNING Sector repair completed: port=0, LBA=0xE9FE2AB<br />
c0 [Wed Oct 31 02:02:03 2012] WARNING Sector repair completed: port=0, LBA=0x1460C25<br />
c0 [Wed Oct 31 02:04:05 2012] WARNING Sector repair completed: port=0, LBA=0x1460C28<br />
c0 [Thu Nov 1 00:48:34 2012] WARNING Sector repair completed: port=1, LBA=0xA7C92BE<br />
c0 [Thu Nov 1 05:04:45 2012] WARNING Sector repair completed: port=0, LBA=0x1C252C2<br />
<br />
[root@backup1 /data/deprecated]#<br />
</pre><br />
<br />
Look for failed drives and errors. Obviously from the above we need to probably replace drives 0 and 1 and drive 1 is even showing as having problems, yet the RAID array is healthy, amazingly. You also see the automatic verifies.<br />
<br />
Note: when rebuilding a degraded mirror, you will see no progress as it rebuilds in the cli<br />
<pre>Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u0 RAID-5 REBUILDING 0 - 64K 4656.56 OFF ON</pre><br />
<br />
<br />
=== areca ===<br />
<br />
We are using an areca controller on backup3. <br />
<br />
<pre><br />
[root@newbackup3 ~]# sh /root/verify.sh<br />
# Name Raid Name Level Capacity Ch/Id/Lun State<br />
===============================================================================<br />
1 ARC-1160-VOL#00 Raid Set # 00 Raid5 5000.0GB 00/00/00 Checking(19.7%)<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
# Name Disks TotalCap FreeCap DiskChannels State<br />
===============================================================================<br />
1 Raid Set # 00 6 6000.0GB 0.0GB 123456 Checking<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
Date-Time Device Event Type Elapsed Time Errors<br />
===============================================================================<br />
2012-12-05 20:40:58 ARC-1160-VOL#00 Start Checking<br />
2012-12-01 05:06:04 ARC-1160-VOL#00 Complete Init 027:30:45<br />
2012-11-30 01:35:19 ARC-1160-VOL#00 Start Initialize<br />
2026-08-06 01:34:52 H/W MONITOR Raid Powered On<br />
2012-11-30 01:33:36 ARC-1160-VOL#00 Stop Initialization 000:31:48<br />
2012-11-30 01:01:47 ARC-1160-VOL#00 Start Initialize<br />
2026-08-06 00:58:13 H/W MONITOR Raid Powered On<br />
2012-11-30 00:57:26 ARC-1160-VOL#00 Stop Initialization 000:57:07<br />
2012-11-30 00:00:19 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 23:56:48 H/W MONITOR Raid Powered On<br />
2026-08-05 23:52:58 H/W MONITOR Raid Powered On<br />
2026-08-05 23:50:14 H/W MONITOR Raid Powered On<br />
2026-08-05 23:43:30 H/W MONITOR Raid Powered On<br />
2012-11-29 23:10:07 ARC-1160-VOL#00 Stop Initialization 000:00:56<br />
2012-11-29 23:09:11 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 23:08:57 H/W MONITOR Raid Powered On<br />
2012-11-29 23:08:10 ARC-1160-VOL#00 Stop Initialization 000:20:41<br />
2012-11-29 22:47:29 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 22:46:59 H/W MONITOR Raid Powered On<br />
2026-08-05 22:45:55 H/W MONITOR Raid Powered On<br />
2026-08-05 22:44:53 H/W MONITOR Raid Powered On<br />
2026-08-05 22:42:06 H/W MONITOR Raid Powered On<br />
2026-08-05 22:40:50 H/W MONITOR Raid Powered On<br />
2012-11-29 22:40:04 ARC-1160-VOL#00 Stop Initialization 000:24:25<br />
2012-11-29 22:15:38 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 22:15:11 000:000001215B00 Restart Init LBA Point<br />
2026-08-05 22:15:10 H/W MONITOR Raid Powered On<br />
2012-11-29 21:56:38 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 21:56:12 H/W MONITOR Raid Powered On<br />
2026-08-05 21:56:04 IDE Channel #03 Device Inserted<br />
2012-11-29 21:55:13 IDE Channel #04 Device Inserted<br />
2012-11-29 21:55:03 IDE Channel #02 Device Inserted<br />
2026-08-05 21:53:09 H/W MONITOR Raid Powered On<br />
2026-08-05 20:51:46 H/W MONITOR Raid Powered On<br />
2026-08-05 20:49:56 H/W MONITOR Raid Powered On<br />
2026-08-05 20:48:29 H/W MONITOR Raid Powered On<br />
2026-08-05 20:46:29 H/W MONITOR Raid Powered On<br />
2026-08-05 20:44:49 H/W MONITOR Raid Powered On<br />
2026-08-05 20:43:01 H/W MONITOR Raid Powered On<br />
2026-08-05 20:36:25 H/W MONITOR Raid Powered On<br />
2026-08-05 20:31:18 H/W MONITOR Raid Powered On<br />
2026-08-05 20:30:08 H/W MONITOR Raid Powered On<br />
2026-08-05 20:08:40 H/W MONITOR Raid Powered On<br />
2026-08-05 20:06:11 H/W MONITOR Raid Powered On<br />
2026-08-05 20:05:14 H/W MONITOR Raid Powered On<br />
2026-08-05 20:03:58 H/W MONITOR Raid Powered On<br />
2026-08-05 20:00:56 H/W MONITOR Raid Powered On<br />
2026-08-05 19:57:57 H/W MONITOR Raid Powered On<br />
2026-08-05 19:56:15 H/W MONITOR Raid Powered On<br />
2026-08-05 19:55:05 H/W MONITOR Raid Powered On<br />
2026-08-05 17:24:36 H/W MONITOR Raid Powered On<br />
2026-08-05 17:22:43 H/W MONITOR Raid Powered On<br />
2026-08-05 04:50:42 H/W MONITOR Raid Powered On<br />
2026-08-05 04:47:33 H/W MONITOR Raid Powered On<br />
2026-08-05 04:43:57 H/W MONITOR Raid Powered On<br />
2026-08-05 04:18:52 H/W MONITOR Raid Powered On<br />
2026-08-05 04:17:30 H/W MONITOR Raid Powered On<br />
2026-08-05 04:13:30 H/W MONITOR Raid Powered On<br />
2026-08-05 04:10:26 H/W MONITOR Raid Powered On<br />
2026-08-05 04:09:23 H/W MONITOR Raid Powered On<br />
2026-08-05 00:08:09 H/W MONITOR Raid Powered On<br />
2026-08-05 00:07:12 H/W MONITOR Raid Powered On<br />
2026-08-05 00:05:51 H/W MONITOR Raid Powered On<br />
2026-08-05 00:04:27 H/W MONITOR Raid Powered On<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
press enter when ready to run verify<br />
</pre><br />
<br />
Look for failed drives and errors. <br />
<br />
When it proceed's to verifying, you can confirm with:<br />
<br />
<pre><br />
[root@newbackup3 ~]# cli64 vsf info<br />
# Name Raid Name Level Capacity Ch/Id/Lun State<br />
===============================================================================<br />
1 ARC-1160-VOL#00 Raid Set # 00 Raid5 5000.0GB 00/00/00 Checking(22.5%)<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
[root@newbackup3 ~]#<br />
</pre><br />
<br />
== Update OS list ==<br />
<br />
# check for any new VZ templates we want to offer: <tt>vzup2date -z</tt><br />
# see if there's any OS's we want to include in our colo install list. Update 2 places: <tt>signup/html/colo_quote.html</tt> & <tt>signup/html/step1.html</tt><br />
# update the mgmt database (ref_templates table, ref_systems table).<br />
<br />
= Infrequent tasks =<br />
<br />
== Free up space on gateway ==<br />
<br />
<pre>newgateway /var/spool# cd clientmqueue/<br />
newgateway /var/spool/clientmqueue# sh<br />
# for f in `ls`; do rm $f; done<br />
exit</pre><br />
<br />
== Free up space on mail ==<br />
<br />
You can clear out root mail:<br />
<pre><br />
mail /var/log# ll -h /var/mail/root<br />
-rw------- 1 root mail 543K Dec 19 13:05 /var/mail/root<br />
mail /var/log# rm /var/mail/root<br />
</pre><br />
<br />
Or you can archive mail logs:<br />
<pre><br />
mail /var/log# ls -l htt*<br />
-rw-r--r-- 1 root wheel 297436931 Dec 19 13:26 httpd-access.log<br />
-rw-r--r-- 1 root wheel 9824324 Jul 4 11:34 httpd-access.log.old.0.gz<br />
-rw-r--r-- 1 root wheel 6884137 Mar 17 2012 httpd-access.log.old.1.gz<br />
-rw-r--r-- 1 root wheel 18557444 Dec 3 2009 httpd-access.log.old.10.gz<br />
-rw-r--r-- 1 root wheel 14740263 Jan 9 2007 httpd-access.log.old.11.gz<br />
-rw-r--r-- 1 root wheel 14209465 Nov 28 2007 httpd-access.log.old.12.gz<br />
-rw-r--r-- 1 root wheel 16874396 Feb 19 2012 httpd-access.log.old.3.gz<br />
-rw-r--r-- 1 root wheel 14554859 Jul 22 2011 httpd-access.log.old.4.gz<br />
-rw-r--r-- 1 root wheel 10513227 Feb 18 2011 httpd-access.log.old.5.gz<br />
-rw-r--r-- 1 root wheel 7201946 Oct 29 2010 httpd-access.log.old.6.gz<br />
-rw-r--r-- 1 root wheel 10062537 May 6 2010 httpd-access.log.old.7.gz<br />
-rw-r--r-- 1 root wheel 10157042 Aug 12 2010 httpd-access.log.old.8.gz<br />
-rw-r--r-- 1 root wheel 11909534 Mar 4 2010 httpd-access.log.old.9.gz<br />
-rw-r--r-- 1 root wheel 59030930 Dec 19 13:01 httpd-error.log<br />
-rw-r--r-- 1 root wheel 3413134 Mar 4 2010 httpd-error.log.0.gz<br />
-rw-r--r-- 1 root wheel 795515 May 1 2007 httpd-error.log.1.gz<br />
-rw-r--r-- 1 root wheel 1142153 Nov 30 2007 httpd-error.log.2.gz<br />
-rw-r--r-- 1 root wheel 2325801 Feb 18 2011 httpd-error.log.gz<br />
<br />
mail /var/log# sh<br />
# for f in 12 11 10 9 8 7 6 5 4 3 2 1 0; do g=`echo $f+1|bc`; mv httpd-access.log.old.$f.gz httpd-access.log.old.$g.gz; done<br />
# mv httpd-access.log httpd-access.log.old.0<br />
# touch httpd-access.log<br />
# apachectl restart<br />
# gzip httpd-access.log.old.0<br />
<br />
# for f in 2 1 0; do g=`echo $f+1|bc`; mv httpd-error.log.$f.gz httpd-error.log.$g.gz; done<br />
# mv httpd-error.log httpd-error.log.0<br />
# touch httpd-error.log<br />
# apachectl restart<br />
# gzip httpd-error.log.0<br />
# exit<br />
</pre><br />
<br />
== Free up space on bwdb2 ==<br />
<br />
You can either remove items from <tt>/usr/home/archive</tt> or you can scp them to backup3:/data/bwdb2/archive .<br />
<br />
== Free up space on backup1 ==<br />
<br />
backup1 is our primary customer backup system. As usage grows over time, it needs to be regularly purged of old files. The easiest way to do this is by removing deprecated files. These mostly consist of cancelled customers or temporary dump/storage files (created during dump/restores). Our standard policy is to hang onto cancelled customers for 6mos after which we remove their files (as far as customers know their data is purged immediately, but we hang onto it just in case.. and in some cases we cancel a server due to non payment so this makes it easy to restore their system). To find files to remove:<br />
<br />
<pre>[root@backup1 ~]# cd /data/deprecated/<br />
[root@backup1 /data/deprecated]# ls<br />
2101-migrated-20120317.tgz old-683-cxld-20121021.tgz<br />
69.55.230.2-wwwbackup old-744-cxld-20120708.tgz<br />
991-DONTDELETE.tgz old-809-cxld-20120609.tgz<br />
archive-col02050-mdfile-cxld-20120409.gz old-854-cxld-20120621.tgz<br />
col01371.tgz old-931-cxld-20060513.tgz<br />
deleteme_ubuntu-10.10-x86_20111205 old-col00123-mdfile-noarchive-20120417.gz<br />
jail10_old old-col00147-vnfile-cxld-20120828.gz<br />
jail14_rsync_old old-col00419-dump-cxld-20120224.gz<br />
jail15_old old-col01098-vnfile-cxld-20120827.gz<br />
jail3_old old-col01278-dump-cxld-20120822<br />
jail4_old old-col01517-dump-cxld-20120828<br />
jail5_old old-col01669-dump-cxld-20120203.gz<br />
old-1009-cxld-20120608.tgz old-col01687-dump-cxld-20120909<br />
old-1012-cxld-20120411.tgz old-col01790-dump-cxld-20120828<br />
old-1052-cxld-20120721.tgz old-col01812-dump-cxld-20120820<br />
old-10631-cxld-20120622.tgz old-col01938-mdfile-cxld-20120619.gz<br />
old-10632-cxld-20120622.tgz old-col02095-mdfile-noarchive-20120523.gz<br />
old-10633-cxld-20120622.tgz olddebian-3.0-v15-20110610.tgz<br />
old-1236-cxld-20120621.tgz oldmod_frontpage-deb30-v15-20110610.tgz<br />
old-1381-cxld-20120404.tgz oldmod_perl-deb30-v15-20110610.tgz<br />
old-1422-cxld-20120721.tgz oldmod_ssl-deb30-v15-20110610.tgz<br />
old-14681-cxld-20120619.tgz oldmysql-deb30-v15-20110610.tgz<br />
old-1544-cxld-20120626.tgz oldproftpd-deb30-v15-20110610.tgz<br />
old-18351-cxld-20120605.tgz old_virt14<br />
old-1853-cxld-20120910.tgz old_virt18<br />
old-1963-cxld-20120206.tgz oldwebmin-deb30-v15-20110610.tgz<br />
old-1967-cxld-20120605.tgz suse.virt11.20120421.tgz<br />
old-1981-noarchive-20120729.tgz virt11<br />
old-2030-migrated-noarchive-20120727.tgz virt12_old<br />
old-2037-cxld-20120716.tgz virt13_old<br />
old-2065-cxld-20120727.tgz virt16_old<br />
old-2068-cxld-20120424.tgz virt4_old<br />
old-2085-cxld-20120531.tgz virt5_old<br />
old-364-cxld-20120904.tgz virt6_old<br />
old-446-cxld-20120512.tgz virt7_old<br />
old-613-cxld-20120601.tgz virt8_old<br />
[root@backup1 /data/deprecated]#<br />
</pre><br />
<br />
virtX_old and jailX_old are permanently archived, so ignore those as well as anything else marked not to delete or otherwise suspicious. Likewise, probably a good idea to try to hang onto oldTEMPLATE.gz as long as we can as well.<br />
Most of the stuff we want to delete is dated when it was deprecated, making this easy. So to remove files from 6 mos ago (running this in Oct):<br />
<br />
<pre>[root@backup1 /data/deprecated]# ls old*201204*<br />
old-1012-cxld-20120411.tgz old-2068-cxld-20120424.tgz<br />
old-1381-cxld-20120404.tgz old-col00123-mdfile-noarchive-20120417.gz<br />
[root@backup1 /data/deprecated]# rm old*201204*<br />
</pre><br />
<br />
Every few months you will also want to remove some of the snapshot archives for mail. We typically save the 1st, 10th, and 20th of each month. To do this you set aside the dates you want to save then remove months at a time, followed by restoring the set aside dates. Here's how that works:<br />
<pre>[root@backup1 /data/www/daily]# ls<br />
05 08-10-11 10-04-10 11-10-10 12-07-29 12-09-21 12-11-14<br />
06 08-10-21 10-04-20 11-10-20 12-07-30 12-09-22 12-11-15<br />
06-06-01-usr-home.tgz 08-11-01 10-05-01 11-11-01 12-07-31 12-09-23 12-11-16<br />
06-07-01-usr-home.tgz 08-11-10 10-05-11 11-11-10 12-08-01 12-09-24 12-11-17<br />
06-08-01-usr-home.tgz 08-11-20 10-05-20 11-11-20 12-08-02 12-09-25 12-11-18<br />
06-09-01-usr-home.tgz 08-12-01 10-06-01 11-12-01 12-08-03 12-09-26 12-11-19<br />
06-11-10 08-12-10 10-06-10 11-12-10 12-08-04 12-09-27 12-11-20<br />
06-12-21 08-12-20 10-06-20 11-12-20 12-08-05 12-09-28 12-11-21<br />
07-01-10 09-01-01 10-07-01 12-01-01 12-08-06 12-09-29 12-11-22<br />
07-01-20 09-01-10 10-07-10 12-01-10 12-08-07 12-09-30 12-11-23<br />
07-02-10 09-01-20 10-07-20 12-01-20 12-08-08 12-10-01 12-11-24<br />
07-02-20 09-02-01 10-08-01 12-02-01 12-08-09 12-10-02 12-11-25<br />
07-03-01 09-02-10 10-08-10 12-02-10 12-08-10 12-10-03 12-11-26<br />
07-03-20 09-02-20 10-08-20 12-02-20 12-08-11 12-10-04 12-11-27<br />
07-04-01 09-03-01 10-09-01 12-03-01 12-08-12 12-10-05 12-11-28<br />
07-04-10 09-03-10 10-09-10 12-03-10 12-08-13 12-10-06 12-11-29<br />
07-04-20 09-03-20 10-09-20 12-03-20 12-08-14 12-10-07 12-11-30<br />
07-05-01 09-04-01 10-10-01 12-04-01 12-08-15 12-10-08 12-12-01<br />
07-05-10 09-04-10 10-10-10 12-04-10 12-08-16 12-10-09 12-12-02<br />
07-05-20 09-04-20 10-10-20 12-04-20 12-08-17 12-10-10 12-12-03<br />
07-06-01 09-05-01 10-11-01 12-05-01 12-08-18 12-10-11 12-12-04<br />
07-06-10 09-05-10 10-11-10 12-05-10 12-08-19 12-10-12 12-12-05<br />
07-06-20 09-05-20 10-11-20 12-05-20 12-08-20 12-10-13 12-12-06<br />
07-07-20 09-06-01 10-12-01 12-06-01 12-08-21 12-10-14 12-12-07<br />
07-08-10 09-06-10 10-12-10 12-06-10 12-08-22 12-10-15 12-12-08<br />
07-08-20 09-06-20 10-12-20 12-06-20 12-08-23 12-10-16 12-12-09<br />
07-09-01 09-07-01 11-01-01 12-07-01 12-08-24 12-10-17 12-12-10<br />
07-10-01 09-07-10 11-01-10 12-07-02 12-08-25 12-10-18 12-12-11<br />
07-10-10 09-07-20 11-01-21 12-07-03 12-08-26 12-10-19 12-12-12<br />
07-10-20 09-08-01 11-02-01 12-07-04 12-08-27 12-10-20 12-12-13<br />
07-12-01 09-08-10 11-02-10 12-07-05 12-08-28 12-10-21 12-12-14<br />
07-12-10 09-08-20 11-02-20 12-07-06 12-08-29 12-10-22 12-12-15<br />
08-01-01 09-09-01 11-03-01 12-07-07 12-08-30 12-10-23 12-12-16<br />
08-01-20 09-09-10 11-03-10 12-07-08 12-08-31 12-10-24 12-12-17<br />
08-02-20 09-09-20 11-03-20 12-07-09 12-09-01 12-10-25 12-12-18<br />
08-03-01 09-10-01 11-04-01 12-07-10 12-09-02 12-10-26 12-12-19<br />
08-03-10 09-10-10 11-04-10 12-07-11 12-09-03 12-10-27 12-12-20<br />
08-03-20 09-10-20 11-04-20 12-07-12 12-09-04 12-10-28 12-12-21<br />
08-04-01 09-11-01 11-05-01 12-07-13 12-09-05 12-10-29 12-12-22<br />
08-04-20 09-11-10 11-05-10 12-07-14 12-09-06 12-10-30 12-12-23<br />
08-05-01 09-11-20 11-05-20 12-07-15 12-09-07 12-10-31 12-12-24<br />
08-05-10 09-12-01 11-06-01 12-07-16 12-09-08 12-11-01 12-12-25<br />
08-06-10 09-12-10 11-06-10 12-07-17 12-09-09 12-11-02 12-12-26<br />
08-06-20 09-12-20 11-06-20 12-07-18 12-09-10 12-11-03 12-12-27<br />
08-07-02 10-01-01 11-07-01 12-07-19 12-09-11 12-11-04 12-12-28<br />
08-07-10 10-01-10 11-07-10 12-07-20 12-09-12 12-11-05 2008-10-23<br />
08-07-20 10-01-20 11-07-20 12-07-21 12-09-13 12-11-06 bb.tgz<br />
08-08-01 10-02-01 11-08-01 12-07-22 12-09-14 12-11-07 boot<br />
08-08-10 10-02-10 11-08-10 12-07-23 12-09-15 12-11-08 current<br />
08-08-21 10-02-20 11-08-20 12-07-24 12-09-16 12-11-09 hold<br />
08-09-01 10-03-01 11-09-01 12-07-25 12-09-17 12-11-10<br />
08-09-10 10-03-10 11-09-10 12-07-26 12-09-18 12-11-11<br />
08-09-21 10-03-20 11-09-20 12-07-27 12-09-19 12-11-12<br />
08-10-01 10-04-01 11-10-01 12-07-28 12-09-20 12-11-13<br />
[root@backup1 /data/www/daily]#<br />
</pre><br />
<br />
So we see that everything up to July 2012 has been pruned. To prune July 2012 we do the following:<br />
<br />
<pre>mv 12-07-01 hold<br />
mv 12-07-10 hold<br />
mv 12-07-20 hold<br />
rm -fr 12-07*<br />
mv hold/* .</pre><br />
<br />
== Free up space on Other Servers ==<br />
<br />
Many servers start to run out of disk space over time. Often it is caused by unread mail for<br />
root or log files. <br />
<br />
To find the source of the problems, you use "du" to find where the disk space is being used.<br />
You can't do a du on /proc or /dev, si I use the command<br />
<br />
<pre><br />
[root@virt11 /]# du -hs [a-c]* deprecated [e-o]* [q-u]* var | tee duhs0<br />
<br />
which produces something like this.<br />
<br />
4.0K backup<br />
4.0K backup1<br />
4.0K backup2<br />
4.0K backup3<br />
4.0K backup4<br />
7.5M bin<br />
47M boot<br />
4.0K deprecated<br />
92M etc<br />
30M home<br />
8.0K initrd<br />
541M lib<br />
16K lost+found<br />
8.0K media<br />
0 misc<br />
8.0K mnt<br />
0 net<br />
92M opt<br />
336K root<br />
36M sbin<br />
8.0K selinux<br />
8.0K srv<br />
0 sys<br />
4.0K test<br />
16K tmp<br />
1.2G usr<br />
583M var<br />
<br />
In this case it looks like /var is the problem, so<br />
<br />
cd /var<br />
du -hs * | tee duhs9<br />
<br />
Produces<br />
<br />
12K account<br />
2.6M analog-5.32<br />
63M cache<br />
24K db<br />
4.0K duhs<br />
4.0K duhs1<br />
4.0K duhs2<br />
4.0K duhs3<br />
4.0K duhs4<br />
4.0K duhs5<br />
4.0K duhs6<br />
4.0K duhs7<br />
4.0K duhs8<br />
32K empty<br />
8.0K games<br />
16K kerberos<br />
42M lib<br />
8.0K local<br />
36K lock<br />
457M log<br />
0 mail<br />
8.0K nis<br />
8.0K opt<br />
8.0K preserve<br />
8.0K racoon<br />
240K run<br />
18M spool<br />
8.0K tmp<br />
64K vz<br />
0 vzagent<br />
0 vzagent.tmp<br />
16K vzquota<br />
1.2M www<br />
20K yp<br />
<br />
Usually, the problem is in /var/spool or /var/log, due to unread mail<br />
or excessive log files. You can continue to drill down by doing<br />
a "cd <sbdirectory>" and another "du -hs *".<br />
<br />
</pre></div>75.80.128.147https://wiki.jcihosting.com/index.php?title=Routine_Maintenance&diff=2231Routine Maintenance2020-06-09T21:59:04Z<p>75.80.128.147: /* Free up space on backup1 */</p>
<hr />
<div>= Daily Tasks =<br />
<br />
== check load graphs == <br />
<br />
Click on the Load link in mgmt <br />
<br />
This screen shows you load levels on our servers and network traffic for critical machines (firewalls, backup servers).<br />
<br />
If you see load high or increasing <br />
<br />
FreeBSD: <br />
run [[VPS_Management#jtop|jtop]] (or [[VPS_Management#jt|jt]] > 7.x) and see if there are any runaway processes. Here are some examples of entries in top that are<br />
definitely runaway processes:<br />
<br />
<pre>79481 root 64 0 2256K 1056K CPU1 1 58:16 87.40% 87.40% nano<br />
50650 1000 64 0 1852K 1112K RUN 0 207.9H 84.08% 84.08% screen<br />
14829 www 2 0 39100K 31736K accept 0 104:24 46.54% 6.54% httpd<br />
42065 root 61 0 1300K 844K RUN 1 47.8H 91.36% 91.36% ee<br />
1328 www 56 0 18440K 10796K CPU1 0 64.4H 97.71% 97.71% httpd<br />
26251 user 57 0 6124K 1160K CPU1 1 82.9H 98.44% 98.44% screen<br />
89874 root 60 0 1352K 892K RUN 1 33.8H 65.82% 65.82% dialog<br />
38656 1000 64 0 3088K 2136K CPU0 0 806:13 97.95% 97.95% StutBot<br />
27630 root 64 0 1396K 972K RUN 1 76.8H 86.47% 86.47% ee</pre><br />
<br />
Linux:<br />
run [[VPS_Management#vwe|vwe]] to see which VPS’s have high loads. From there run <tt>[[VPS_Management#vp|vp]] <veid></tt> and/or <tt>[[VPS_Management#vt|vt]] <veid></tt> to see what's going on in that system.<br />
[[VPS_Management#vzstat|vzstat]] will also give you a nice picture of whats going on, systems with high numbers in the mlat column are likely culprits.<br />
<br />
examples of out of control procs:<br />
<br />
<pre>12183 nobody 16 0 4916 1348 1340 R 45.5 0.0 4249m httpd<br />
29266 #502 16 0 1852 796 792 R 22.5 0.0 1104m vim<br />
23860 #41 16 0 5472 5472 2076 R 98.9 0.2 31:41 python<br />
19227 bin 19 0 1688 716 652 R 99.9 0.0 321:08 wtrs_ui<br />
7762 apache 16 0 268 236 224 R 85.7 0.0 1010m ptrace<br />
4624 #501 20 0 4304 2400 2044 R 53.6 0.1 284:32 YoSucker<br />
20451 #506 20 0 1876 820 816 R 17.2 0.0 169:35 vim<br />
8834 #514 20 0 900 724 672 R 77.6 0.0 382:30 neostats<br />
31815 apache 14 0 3176 3176 1696 R 74.4 0.1 6:15 counter</pre><br />
<br />
Just kill -9 them and be done with it.<br />
<br />
Also, anytime you see `kmod` or `ptrace` - kill those immediaely no matter how much they are using - they are attempts to exploit the linux ptrace bug. They won't work, but they suck a lot of CPU...<br />
<br />
Also, any other processes that are at 90-100% cpu usage and have been running for any long period of time should be killed except for mysqld processes on FreeBSD. See above.<br />
<br />
However, there is an exception:<br />
<br />
if it is a mysqld, we don't want to kill their database. What you want to do is <tt>[[VPS_Management#jpid|jpid]] <pid></tt> to see who owns it, and then email them the paste containing the instructions for the nanny. Or you can simply do a <tt>kill -1 PID</tt> on the process to restart it.<br />
<br />
=== Load averages jump at night ===<br />
<br />
The load averages on the FreeBSD systems may jump up at night between 1 and 4 am - this is because the backups are running - if this is what is causing a jump in load, you will see processes like `rsync` in top eating a lot of CPU time.<br />
<br />
== check backups ==<br />
<br />
mgmt -> Motnroing -> Backups and make sure every machine was backed up the previous nite. <br />
Also look at df on backup1 and backup2 to make sure no disk is approaching full, though bb should warn us in advance. Please note - errors encountered when a backup script on any of the particular systems run will gnerate an email to support@johncompanies.com so you can know immediately the day after if the directory to be backed up has been moved or no longer exists. A paste exists for this to notify the customer of a non-existant file/dir.<br />
<br />
== check bb for warnings ==<br />
mgmt -> BigBrother<br />
<br />
Some events don't generate pages (on purpose). You will only see them by going to the bb main page.<br />
<br />
== check jail5 for crashed VPSs ==<br />
<br />
On jail5<br />
<br />
notrunning<br />
<br />
To restart a VPS <br />
<br />
vm restart col0XXXX<br />
<br />
== Check NetHere ==<br />
<br />
Check the NetHere servers. To get into the servers, login to<br />
admin-1.nethere.net and su - to root.<br />
<br />
=== Mail systems ===<br />
Check for possible SPAMMERs.<br />
<br />
==== Incoming ====<br />
Check mta-1 and mta-2 count of customer logins for possible<br />
customer SPAM compromises.<br />
<br />
login_count /logs/maillog | tail -30<br />
<br />
==== Outgoing ====<br />
<br />
Check outgoing queues on relay-1 and relay-2<br />
<br />
mail_count | tail -30<br />
<br />
To clean up outgoing queues of unwanted SPAM on relay-1 and relay-2.<br />
<br />
mail_cleanup [ <sender's domain/username/message id> ... ]<br />
<br />
To just remove emails from some senders.<br />
<br />
rmmails <sender's domain/username/message id> ...<br />
<br />
=== Nagios ===<br />
<br />
Check for other problems on NetHere servers<br />
<br />
https://nagios.nethere.net<br />
<br />
=== Cacti ===<br />
<br />
Check bandwidth usage on servers<br />
<br />
https://cacti.nethere.net<br />
<br />
= Monthly Tasks =<br />
== rotate pine sent mail (1st of month) ==<br />
On the 1st of the month, before any emails are sent out, quit out of pine, then log back in. Send mail from last month will be archived.<br />
If you mess up and do it on the 3rd (for example), you can go into the previous month's saved email and save emails from the current month into the <tt>sent-mail</tt> (current month) mailbox.<br />
<br />
== b/w caps ==<br />
On the 1st: remove any bwcaps put into the firewall (only really applies if a bwcap was added cause someone went over on b/w):<br />
<br />
ipfw list|grep pipe<br />
ipfw del [each rule listed]<br />
<br />
NOTE: this cronjob on newgateway will do some of that for you, provided you used one of the following pipe #s:<br />
0 0 1 * * /sbin/ipfw del 3 4 5 17331<br />
<br />
-------------------<br />
<br />
We really don’t do this anymore since we have centralized traffic accounting with netflow, but for posterity:<br />
<br />
Make sure all machines reset counters to 0 after midnight on the 1st<br />
Make sure they dumped a counter<br />
<br />
On each jail run:<br />
trafficgather.pl<br />
<br />
And on each virt:<br />
linuxtrafficgather.pl<br />
<br />
== Monthly RAID checks ==<br />
<br />
Every month we check the health of and verfy the parity on all our RAID-based systems.<br />
To facilitate this, we've created a simple script to start the process:<br />
<br />
sh /root/verify.sh<br />
<br />
=== Adaptec controllers ===<br />
Here's some sample output:<br />
<pre><br />
mail /usr/local/www/scripts# sh /root/verify.sh<br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> container list /f<br />
Executing: container list /full=TRUE<br />
Num Total Oth Chunk Scsi Partition<br />
Creation System<br />
Label Type Size Ctr Size Usage B:ID:L Offset:Size State RO Lk Task Done% Ent<br />
Date Time Files<br />
----- ------ ------ --- ------ ------- ------ ------------- ------- -- -- ------- ------ ---<br />
------ -------- ------<br />
0 Mirror 33.9GB Open 0:01:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:32<br />
/dev/aacd0 mirror0 0:00:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:32<br />
<br />
1 Mirror 33.9GB Open 0:02:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:50<br />
/dev/aacd1 mirror1 0:03:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:50<br />
<br />
<br />
AAC0> disk list /f<br />
Executing: disk list /full=TRUE<br />
<br />
B:ID:L Device Type Removable media Vendor-ID Product-ID Rev Blocks Bytes/Bl<br />
ock Usage Shared Rate<br />
------ -------------- --------------- --------- ---------------- ----- --------- --------<br />
--- ---------------- ------ ----<br />
0:00:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:01:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:02:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:03:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
<br />
AAC0> disk show smart<br />
Executing: disk show smart<br />
<br />
Smart Method of Enable<br />
Capable Informational Exception Performance Error<br />
B:ID:L Device Exceptions(MRIE) Control Enabled Count<br />
------ ------- ---------------- --------- ----------- ------<br />
0:00:0 Y 6 Y N 0<br />
0:01:0 Y 6 Y N 0<br />
0:02:0 Y 6 Y N 0<br />
0:03:0 Y 6 Y N 0<br />
0:06:0 N<br />
<br />
AAC0> task list<br />
Executing: task list<br />
<br />
Controller Tasks<br />
<br />
TaskId Function Done% Container State Specific1 Specific2<br />
------ -------- ------- --------- ----- --------- ---------<br />
<br />
No tasks currently running on controller<br />
<br />
AAC0> dia sh hi<br />
Executing: diagnostic show history<br />
No switches specified, defaulting to "/current".<br />
<br />
<br />
<br />
*** HISTORY BUFFER FROM CURRENT CONTROLLER RUN ***<br />
<br />
[00]: GetDiskLogEntry: container - 1, entry return 0<br />
[01]: Container 1 started SCRUB task<br />
[02]: Starting Mirror:1 scrub<br />
[03]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[04]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[05]: UpdateDiskLogIndex - Set - container 0, index 1<br />
[06]: GetDiskLogEntry: container - 0, entry return 1<br />
[07]: Container 0 started SCRUB task<br />
[08]: Starting Mirror:0 scrub<br />
[09]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[10]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[11]: Mirror Scrub Container:1 ErrorsFound:0<br />
[12]: Clear disk log: sector - 80, driveno 2<br />
[13]: Clear disk log: sector - 80, driveno 3<br />
[14]: Container 1 completed SCRUB task:<br />
[15]: Mirror Scrub Container:0 ErrorsFound:0<br />
[16]: Clear disk log: sector - 81, driveno 1<br />
[17]: Clear disk log: sector - 81, driveno 0<br />
[18]: Container 0 completed SCRUB task:<br />
[19]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[20]: GetDiskLogEntry: container - 0, entry return 0<br />
[21]: Container 0 started SCRUB task<br />
[22]: Starting Mirror:0 scrub<br />
[23]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[24]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[25]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[26]: GetDiskLogEntry: container - 1, entry return 1<br />
[27]: Container 1 started SCRUB task<br />
[28]: Starting Mirror:1 scrub<br />
[29]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[30]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[31]: Mirror Scrub Container:1 ErrorsFound:0<br />
[32]: Clear disk log: sector - 81, driveno 2<br />
[33]: Clear disk log: sector - 81, driveno 3<br />
[34]: Container 1 completed SCRUB task:<br />
[35]: Mirror Scrub Container:0 ErrorsFound:0<br />
[36]: Clear disk log: sector - 80, driveno 1<br />
[37]: Clear disk log: sector - 80, driveno 0<br />
[38]: Container 0 completed SCRUB task:<br />
[39]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[40]: GetDiskLogEntry: container - 0, entry return 0<br />
[41]: Container 0 started SCRUB task<br />
[42]: Starting Mirror:0 scrub<br />
[43]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[44]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[45]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[46]: GetDiskLogEntry: container - 1, entry return 1<br />
[47]: Container 1 started SCRUB task<br />
[48]: Starting Mirror:1 scrub<br />
[49]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[50]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[51]: Mirror Scrub Container:1 ErrorsFound:0<br />
[52]: Clear disk log: sector - 81, driveno 2<br />
[53]: Clear disk log: sector - 81, driveno 3<br />
[54]: Container 1 completed SCRUB task:<br />
[55]: Mirror Scrub Container:0 ErrorsFound:0<br />
[56]: Clear disk log: sector - 80, driveno 1<br />
[57]: Clear disk log: sector - 80, driveno 0<br />
[58]: Container 0 completed SCRUB task:<br />
[59]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[60]: GetDiskLogEntry: container - 0, entry return 0<br />
[61]: Container 0 started SCRUB task<br />
[62]: Starting Mirror:0 scrub<br />
[63]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[64]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[65]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[66]: GetDiskLogEntry: container - 1, entry return 1<br />
[67]: Container 1 started SCRUB task<br />
[68]: Starting Mirror:1 scrub<br />
[69]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[70]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[71]: Mirror Scrub Container:1 ErrorsFound:0<br />
[72]: Clear disk log: sector - 81, driveno 2<br />
[73]: Clear disk log: sector - 81, driveno 3<br />
[74]: Container 1 completed SCRUB task:<br />
[75]: Mirror Scrub Container:0 ErrorsFound:0<br />
[76]: Clear disk log: sector - 80, driveno 1<br />
[77]: Clear disk log: sector - 80, driveno 0<br />
[78]: Container 0 completed SCRUB task:<br />
[79]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[80]: GetDiskLogEntry: container - 0, entry return 0<br />
[81]: Container 0 started SCRUB task<br />
[82]: Starting Mirror:0 scrub<br />
[83]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[84]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[85]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[86]: GetDiskLogEntry: container - 1, entry return 1<br />
[87]: Container 1 started SCRUB task<br />
[88]: Starting Mirror:1 scrub<br />
[89]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[90]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[91]: Mirror Scrub Container:1 ErrorsFound:0<br />
[92]: Clear disk log: sector - 81, driveno 2<br />
[93]: Clear disk log: sector - 81, driveno 3<br />
[94]: Container 1 completed SCRUB task:<br />
[95]: Mirror Scrub Container:0 ErrorsFound:0<br />
[96]: Clear disk log: sector - 80, driveno 1<br />
[97]: Clear disk log: sector - 80, driveno 0<br />
[98]: Container 0 completed SCRUB task:<br />
[99]:<br />
<br />
========================<br />
History Output Complete.<br />
<br />
AAC0><br />
AAC0> exit<br />
Executing: exit<br />
<br />
press enter when ready to run verify <INS><br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> contai scr 0<br />
Executing: container scrub 0<br />
<br />
AAC0> contai scr 1<br />
Executing: container scrub 1<br />
<br />
AAC0> exit<br />
Executing: exit<br />
<br />
when done run: <br />
<br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
<br />
<br />
Nov 1 10:32:46 mail /kernel: aac0: **Monitor** Container 0 started SCRUB task<br />
Nov 1 10:32:47 mail /kernel: aac0: **Monitor** Container 1 started SCRUB task<br />
<br />
</pre><br />
<br />
Here's an analysis of what we're seeing and what we're looking for:<br />
<br />
<pre><br />
AAC0> container list /f<br />
Executing: container list /full=TRUE<br />
Num Total Oth Chunk Scsi Partition<br />
Creation System<br />
Label Type Size Ctr Size Usage B:ID:L Offset:Size State RO Lk Task Done% Ent<br />
Date Time Files<br />
----- ------ ------ --- ------ ------- ------ ------------- ------- -- -- ------- ------ ---<br />
------ -------- ------<br />
0 Mirror 33.9GB Open 0:01:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:32<br />
/dev/aacd0 mirror0 0:00:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:32<br />
<br />
1 Mirror 33.9GB Open 0:02:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:50<br />
/dev/aacd1 mirror1 0:03:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:50<br />
</pre><br />
<br />
This is showing you the health of the arrays. You're looking for ''Normal'' under the State column, and the absence of a ! in the sector size - sometimes, you'll see this:<br />
64.0KB!33.9GB <br />
<br />
That indicates a problem.<br />
<br />
<pre><br />
AAC0> disk show smart<br />
Executing: disk show smart<br />
<br />
Smart Method of Enable<br />
Capable Informational Exception Performance Error<br />
B:ID:L Device Exceptions(MRIE) Control Enabled Count<br />
------ ------- ---------------- --------- ----------- ------<br />
0:00:0 Y 6 Y N 0<br />
0:01:0 Y 6 Y N 0<br />
0:02:0 Y 6 Y N 0<br />
0:03:0 Y 6 Y N 0<br />
0:06:0 N<br />
</pre><br />
<br />
This shows you a SMART report output. Looking for values in the Error Count column.<br />
<pre><br />
AAC0> task list<br />
Executing: task list<br />
<br />
Controller Tasks<br />
<br />
TaskId Function Done% Container State Specific1 Specific2<br />
------ -------- ------- --------- ----- --------- ---------<br />
<br />
No tasks currently running on controller<br />
</pre><br />
<br />
Look for absence of tasks running- a bad thing would be to see a rebuild or verify running when you didn't initiate it.<br />
<br />
With the history output, you're looking for any anomalies or events since the last time a verify was run. If you see a drive with lots of problems, you may want to take backups before allowing the verify to run since it could replicate errors onto the good drive.<br />
<br />
After you see the history output, it will prompt you to press enter to run the verify. If you're happy with all the output you're seeing- mirror is healthy, history looks good, it's safe to proceed. Otherwise ^C to exit. After hitting enter it will start the verify and start to tail the messages log file (so you can easily see when the verify is complete). Here's what that'll look like:<br />
<br />
<pre><br />
Nov 1 14:38:08 mail /kernel: aac0: **Monitor** Container 1 completed SCRUB task:<br />
Nov 1 14:46:45 mail /kernel: aac0: **Monitor** Container 0 completed SCRUB task:<br />
</pre><br />
<br />
So, putting it all together, after hitting enter to start the verify, you'll see:<br />
<br />
<pre><br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> contai scr 0<br />
Executing: container scrub 0<br />
<br />
AAC0> contai scr 1<br />
Executing: container scrub 1<br />
<br />
AAC0> exit<br />
Executing: exit<br />
<br />
when done run: <br />
<br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
<br />
<br />
Nov 1 10:32:46 mail /kernel: aac0: **Monitor** Container 0 started SCRUB task<br />
Nov 1 10:32:47 mail /kernel: aac0: **Monitor** Container 1 started SCRUB task<br />
<br />
</pre><br />
<br />
When the scrub(s) (verify) are complete - if the server has multiple logical drives, it will run both in parallel - you should exit the tail of the log file (^C) and run:<br />
<br />
<pre><br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
</pre><br />
<br />
Which will show you the diagnostic history, you're looking for the results of the most recent scrub:<br />
<br />
<pre><br />
[100]: Mirror Scrub Container:1 ErrorsFound:0<br />
[101]: Clear disk log: sector - 81, driveno 2<br />
[102]: Clear disk log: sector - 81, driveno 3<br />
[103]: Container 1 completed SCRUB task:<br />
[104]: Mirror Scrub Container:0 ErrorsFound:0<br />
[105]: Clear disk log: sector - 80, driveno 1<br />
[106]: Clear disk log: sector - 80, driveno 0<br />
[107]: Container 0 completed SCRUB task:<br />
</pre><br />
<br />
^C to exit the RAID CLI.<br />
<br />
If you see:<br />
<br />
[104]: Mirror Scrub Container:0 ErrorsFound:5<br />
<br />
You'll want to rerun the verify on that drive till it shows 0, or perhaps replace the drive- you should be able to see from the output which drive had the problem.<br />
<br />
Depending on the size and how busy the drive is, the verify can take anywhere from an hour to the better part of a day.<br />
<br />
You will notice that the diagnostic history is not shown on our modern adaptec cards (i.e. any adaptec card not in a Dell 2450). The reason for this is the history is never cleared, so there's simply too much data to show and it just crashes the CLI. So, don't bother trying to see it...which does make it hard to see if there are problems going on, so you just need to watch the scrub to see it goes to 100%. You will also notice that on some servers there's no tail of messages. Again, this is cause no data is shown there about the completion of the scrub. The thing to do here is to go into the CLI and continue to <tt>show tasks</tt> to monitor scrub progress.<br />
<br />
See [[RAIC_CLI#Adaptec|Adaptec RAID CLI Reference]] for more details on how to use the CLI<br />
<br />
=== DELL (LSI-based) SAS controllers ===<br />
<br />
Here's what the output looks like when running verify.sh on a LSI-based card:<br />
<br />
<pre><br />
jail2 /mnt/data2# sh /root/verify.sh<br />
<br />
Adapter #0<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 0<br />
Device Id: 0<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018396142<br />
SAS Address(1): 0x0<br />
Connected Port Number: 0(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DESN<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 1<br />
Device Id: 1<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018395db2<br />
SAS Address(1): 0x0<br />
Connected Port Number: 1(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DERV<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 2<br />
Device Id: 2<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50006eece89<br />
SAS Address(1): 0x0<br />
Connected Port Number: 2(path0)<br />
Inquiry Data: SEAGATE ST3300555SS T2113LM4BFBZ<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 3<br />
Device Id: 3<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50006eee035<br />
SAS Address(1): 0x0<br />
Connected Port Number: 3(path0)<br />
Inquiry Data: SEAGATE ST3300555SS T2113LM4BGF7<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 4<br />
Device Id: 4<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50004bd7ea5<br />
SAS Address(1): 0x0<br />
Connected Port Number: 4(path0)<br />
Inquiry Data: SEAGATE ST3300656SS HS093QP0G8SW<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 5<br />
Device Id: 5<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e01f1c4112<br />
SAS Address(1): 0x0<br />
Connected Port Number: 5(path0)<br />
Inquiry Data: FUJITSU MBA3300RC D306BJ15P9201W06<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 0 (Target Id: 0)<br />
Name:<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:139392MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 1 (Target Id: 1)<br />
Name:MIRROR1<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 2 (Target Id: 2)<br />
Name:MIRROR2<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
Battery FRU : N/A<br />
Battery Warning : Enabled<br />
Memory Correctable Errors : 0<br />
Memory Uncorrectable Errors : 0<br />
BBU : Present<br />
BBU : Yes<br />
Cache When BBU Bad : Disabled<br />
press enter when ready to run verify<br />
<br />
</pre><br />
<br />
Before pressing enter, here's what we're looking for:<br />
<br />
<pre><br />
Enclosure Device ID: 32<br />
Slot Number: 0<br />
Device Id: 0<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018396142<br />
SAS Address(1): 0x0<br />
Connected Port Number: 0(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DESN<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
</pre><br />
<br />
This is the output shown for each physical drive in the system. We're looking to confirm it's Firmware state is ''Online'', and Media Error Count, Other Error Count, and Predictive Failure Count are all zero (or near zero).<br />
<br />
<pre><br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 1 (Target Id: 1)<br />
Name:MIRROR1<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
</pre><br />
<br />
This is the output for each logical drive. We're looking for State ''Optimal''. Also confirm <tt>Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU</tt><br />
<br />
<pre><br />
Exit Code: 0x00<br />
Battery FRU : N/A<br />
Battery Warning : Enabled<br />
Memory Correctable Errors : 0<br />
Memory Uncorrectable Errors : 0<br />
BBU : Present<br />
BBU : Yes<br />
Cache When BBU Bad : Disabled<br />
</pre><br />
<br />
Confirm that the battery is present and error-free.<br />
<br />
If all that checks out, you're ready to proceed with the verify. After pressing enter, the verify is started and here's what you see:<br />
<pre><br />
Start Check Consistency on Virtual Drive 0 (target id: 0) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Start Check Consistency on Virtual Drive 1 (target id: 1) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Start Check Consistency on Virtual Drive 2 (target id: 2) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Check Consistency<br />
<br />
Progress of Virtual Drives...<br />
<br />
Virtual Drive # Percent Complete Time Elps<br />
0 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:03<br />
1 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:02<br />
2 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:01<br />
<br />
Press <ESC> key to quit...<br />
</pre><br />
<br />
The progress for each drive is displayed until all drives have completed the verify. We just want to make sure that each drive goes to completion. No followup is needed...though there probably is a log or history where we can get more info.<br />
<br />
You will notice that jail7 does not run a verify- that's on purpose. The last time we tried this it crashed the system. So, this must be run from the BIOS (take the system offline for a couple hours).<br />
<br />
See [[RAIC_CLI#LSI|LSI RAID CLI Reference]] for more details on how to use the CLI<br />
<br />
=== LSI-based controllers (megaraid) ===<br />
<br />
There is a CLI for this however it's easier to do this with a curses GUI app: megaraid<br />
<br />
Currently only on these servers: virt15, virt16, and firewall2<br />
<br />
To run:<br />
<br />
# cd /usr/local/sbin/; megamgr<br />
<br />
Main menu:<br />
<pre>²ÚÄÄManagement MenuÄÄ¿²<br />
²³ Configure ³²<br />
²³ Initialize ³²<br />
²³ Objects ³²<br />
²³ Rebuild ³²<br />
²³ Check Consistency ³²<br />
²³ Advanced Menu ³²<br />
²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²</pre><br />
<br />
Before you check consistency, make sure the arrays are healthy.<br />
<br />
Objects -> Physical Drive<br />
<br />
Then look to make sure they're all <tt>ONLIN</tt><br />
<pre><br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄObjects - PHYSICAL DRIVE SELECTION MENUÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²<br />
²²²²²²²²³ ³²²²²²²<br />
²²²²²²²²³ Channel-1 ³²²²²²²<br />
²ÚÄÄMana³ ID ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»x ³²²²²²²<br />
²³ Confi³ 0º* ONLIN A01-01º ³²²²²²²<br />
²³ Initi³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²³ Objec³ 1º* ONLIN A01-02º ³²²²²²²<br />
²³ Rebui³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²³ Check³ 2º* ONLIN A02-01º ³²²²²²²<br />
²³ Advan³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²ÀÄÄÄÄÄij 3º* ONLIN A02-02º ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 4º* ONLIN A03-01º ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 5º* ONLIN A03-02ºþ ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 6º* º ³²²²²²²<br />
²²²²²²²²³ ÈÍÍÍÍÍÍÍÍÍÍÍÍÍͼx ³²²²²²²<br />
²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²<br />
²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²²²<br />
²²²²²²²²³Ch-1 ID-5 DISK 140013MB SEAGATE ST3146707LC 0003 ³²²²²²²²²<br />
²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²<br />
</pre><br />
<br />
<br />
Once that's done, hit escape once then the back arror to move back to the Objects menu.<br />
So you select Objects -> Logical Drive -> Logical Drive 1 -> Check Consistency -> YES<br />
<br />
<pre><br />
<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÚÄLogical Drives(02)Ä¿²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ Logical Drive 1 ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²ÚÄÄÄÄObjectsÄÄij Logical Drive 2 ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²ÚÄÄManagemen³ Adapter ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Configure ³ Logical Drive ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Initialize³ Physical Drive ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Objects ³ Channel ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Rebuild ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Check Consistency ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Advanced Menu ³²²²²²²²ÚÄÄÄÄLogical Drive 1ÄÄÄÄÄ¿²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²³ Initialize ÚÄCheck Consistency-1 ?Ä¿²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ Check Consiste³ YES ³²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ View/Update Pa³ NO ³²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³Select YES Or NO³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
</pre><br />
<br />
Then watch the progress. When done, escape back to Logical Drive then repeat for Logical Drive 2. If you ^C or accidentally escape out, you can come back in running the same commands and watch the progress again (it won't restart).<br />
<br />
You can exit megamgr by escaping out or ^C<br />
<br />
=== 3ware ===<br />
<br />
We are using 3ware controllers on backup1 & backup2. Running the verify script will give you different output based on the type of controller:<br />
<br />
<pre><br />
backup2 /d2# sh /root/verify.sh<br />
Controller: c0<br />
-------------<br />
Driver: 1.50.01.002<br />
Model: 7500-8<br />
FW: FE7X 1.05.00.068<br />
BIOS: BE7X 1.08.00.048<br />
Monitor: ME7X 1.01.00.040<br />
Serial #: F11605A3180172<br />
PCB: Rev3<br />
PCHIP: 1.30-33<br />
ACHIP: 3.20<br />
<br />
<br />
# of units: 3<br />
Unit 0: JBOD 186.31 GB ( 390721968 blocks): OK<br />
Unit 1: RAID 5 465.77 GB ( 976790016 blocks): DEGRADED<br />
Unit 5: RAID 5 698.65 GB ( 1465185024 blocks): DEGRADED<br />
<br />
# of ports: 8<br />
Port 0: WDC WD2000JB-00KFA0 WD-WCAMT1451690 186.31 GB (390721968 blocks): OK(unit 0)<br />
Port 1: WDC WD2500JB-00GVC0 WD-WCAL78219488 232.88 GB (488397168 blocks): OK(unit 1)<br />
Port 2: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT)<br />
Port 3: WDC WD2500JB-00GVC0 WD-WMAL73882417 232.88 GB (488397168 blocks): OK(unit 1)<br />
Port 4: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT)<br />
Port 5: WDC WD2500JB-00GVA0 WD-WMAL71338097 232.88 GB (488397168 blocks): OK(unit 5)<br />
Port 6: WDC WD2500JB-32EVA0 WD-WMAEH1301595 232.88 GB (488397168 blocks): OK(unit 5)<br />
Port 7: WDC WD2500JB-00GVC0 WD-WCAL78165566 232.88 GB (488397168 blocks): OK(unit 5)<br />
Controller: c1<br />
-------------<br />
Driver: 1.50.01.002<br />
Model: 7500-8<br />
FW: FE7X 1.05.00.068<br />
BIOS: BE7X 1.08.00.048<br />
Monitor: ME7X 1.01.00.040<br />
Serial #: F11605A3180167<br />
PCB: Rev3<br />
PCHIP: 1.30-33<br />
ACHIP: 3.20<br />
<br />
<br />
# of units: 2<br />
Unit 0: RAID 5 698.65 GB ( 1465185024 blocks): OK<br />
Unit 4: RAID 5 698.65 GB ( 1465185024 blocks): OK<br />
<br />
# of ports: 8<br />
Port 0: WDC WD2500JB-00GVA0 WD-WMAL71301258 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 1: WDC WD2500JB-00GVA0 WD-WMAL71322705 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 2: WDC WD2500JB-00GVA0 WD-WMAL71945050 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 3: WDC WD2500JB-00GVA0 WD-WMAL71316201 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 4: WDC WD2500JB-00GVC0 WD-WCAL78323749 232.88 GB (488397168 blocks): OK(unit 4)<br />
Port 5: WDC WD3200AAJB-00J3A0 WD-WCAV2V689068 298.09 GB (625142448 blocks): OK(unit 4)<br />
Port 6: WDC WD2500JB-00GVC0 WD-WCAL78234420 232.88 GB (488397168 blocks): OK(unit 4)<br />
Port 7: WDC WD2500JB-00GVC0 WD-WCAL78592213 232.88 GB (488397168 blocks): OK(unit 4)<br />
backup2 /d2#<br />
</pre><br />
<br />
On backup2 look for all ok, no verify.<br />
<br />
<pre><br />
[root@backup3 ~]# sh /root/verify.sh<br />
/c2 Driver Version = 1.26.02.002<br />
/c2 Model = 8006-2LP<br />
/c2 Available Memory = 512KB<br />
/c2 Firmware Version = FE8S 1.05.00.068<br />
/c2 Bios Version = BE7X 1.08.00.048<br />
/c2 Boot Loader Version = ME7X 1.01.00.040<br />
/c2 Serial Number = L018501C6481395<br />
/c2 PCB Version = Rev5<br />
/c2 PCHIP Version = 1.30-66<br />
/c2 ACHIP Version = 3.20<br />
/c2 Total Optimal Units = 1<br />
/c2 Not Optimal Units = 0<br />
<br />
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u1 RAID-1 OK - - - 931.512 ON -<br />
<br />
Port Status Unit Size Blocks Serial<br />
---------------------------------------------------------------<br />
p0 OK u1 931.51 GB 1953525168 WD-WMAW31148820<br />
p1 OK u1 931.51 GB 1953525168 WD-WCATR0277515<br />
<br />
<br />
Ctl Date Severity Alarm Message<br />
------------------------------------------------------------------------------<br />
<br />
Sending start verify message to /c2/u1 ... Done.<br />
<br />
<br />
when done run:<br />
tw_cli /c2 show alarms<br />
<br />
[root@backup3 ~]#<br />
</pre><br />
<br />
Automatically starts the verify, just run <tt>tw_cli /c2 show alarms</tt> as instructed to see the results of the verify.<br />
<br />
<pre><br />
[root@backup1 /data/deprecated]# sh /root/verify.sh<br />
/c0 Driver Version = 2.26.02.010<br />
/c0 Model = 9650SE-8LPML<br />
/c0 Available Memory = 224MB<br />
/c0 Firmware Version = FE9X 4.06.00.004<br />
/c0 Bios Version = BE9X 4.05.00.015<br />
/c0 Boot Loader Version = BL9X 3.08.00.001<br />
/c0 Serial Number = L326025A8270177<br />
/c0 PCB Version = Rev 032<br />
/c0 PCHIP Version = 2.00<br />
/c0 ACHIP Version = 1.90<br />
/c0 Number of Ports = 8<br />
/c0 Number of Drives = 6<br />
/c0 Number of Units = 1<br />
/c0 Total Optimal Units = 1<br />
/c0 Not Optimal Units = 0<br />
/c0 JBOD Export Policy = off<br />
/c0 Disk Spinup Policy = 1<br />
/c0 Spinup Stagger Time Policy (sec) = 1<br />
/c0 Auto-Carving Policy = off<br />
/c0 Auto-Carving Size = 2048 GB<br />
/c0 Auto-Rebuild Policy = on<br />
/c0 Controller Bus Type = PCIe<br />
/c0 Controller Bus Width = 1 lane<br />
/c0 Controller Bus Speed = 2.5 Gbps/lane<br />
<br />
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u0 RAID-5 OK - - 64K 4656.56 ON ON<br />
<br />
Port Status Unit Size Blocks Serial<br />
---------------------------------------------------------------<br />
p0 OK u0 931.51 GB 1953525168 9QJ1Y017<br />
p1 DEVICE-ERROR u0 931.51 GB 1953525168 9QJ1ZN07<br />
p2 OK u0 931.51 GB 1953525168 9QJ2XK1R<br />
p3 OK u0 931.51 GB 1953525168 9QJ2010B<br />
p4 OK u0 1.36 TB 2930277168 6XW0L36T<br />
p5 OK u0 931.51 GB 1953525168 WD-WMATV2444836<br />
p6 NOT-PRESENT - - - -<br />
p7 NOT-PRESENT - - - -<br />
<br />
<br />
Ctl Date Severity Alarm Message<br />
------------------------------------------------------------------------------<br />
c0 [Sat May 12 11:27:15 2012] WARNING Sector repair completed: port=0, LBA=0x6AE571C<br />
c0 [Sat May 12 19:16:21 2012] WARNING Sector repair completed: port=1, LBA=0x40E62A23<br />
c0 [Sat May 12 21:40:56 2012] INFO Verify completed: unit=0<br />
c0 [Mon May 14 00:53:53 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFA7<br />
c0 [Mon May 14 00:58:21 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAA<br />
c0 [Mon May 14 04:35:13 2012] WARNING Sector repair completed: port=0, LBA=0x8FEF2CF<br />
c0 [Mon May 14 04:38:22 2012] WARNING Sector repair completed: port=0, LBA=0x8FEF2D1<br />
c0 [Tue May 15 22:53:46 2012] WARNING Sector repair completed: port=0, LBA=0x13C2622<br />
c0 [Wed May 16 00:39:31 2012] WARNING Sector repair completed: port=0, LBA=0x365A67F<br />
c0 [Wed May 16 00:39:37 2012] WARNING Sector repair completed: port=0, LBA=0x365A685<br />
c0 [Wed May 16 00:47:18 2012] WARNING Sector repair completed: port=0, LBA=0x365A687<br />
c0 [Sat May 19 00:01:44 2012] INFO Verify started: unit=0<br />
c0 [Sat May 19 04:46:20 2012] WARNING Sector repair completed: port=0, LBA=0x365A68E<br />
c0 [Sat May 19 13:37:06 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAC<br />
c0 [Sat May 19 13:37:28 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAE<br />
c0 [Sat May 19 13:37:47 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFB1<br />
c0 [Sat May 19 13:38:00 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFB3<br />
c0 [Sat May 19 21:47:45 2012] INFO Verify completed: unit=0<br />
c0 [Wed May 23 12:21:41 2012] INFO Cache synchronization completed: unit=0<br />
c0 [Fri May 25 00:08:19 2012] WARNING Sector repair completed: port=0, LBA=0x12DA76C<br />
c0 [Fri May 25 00:08:34 2012] WARNING Sector repair completed: port=0, LBA=0x12E4901<br />
c0 [Fri May 25 00:09:33 2012] WARNING Sector repair completed: port=0, LBA=0x12DA773<br />
c0 [Fri May 25 00:39:12 2012] WARNING Sector repair completed: port=0, LBA=0x42C597B<br />
c0 [Sat May 26 00:01:45 2012] INFO Verify started: unit=0<br />
c0 [Sat May 26 00:42:05 2012] WARNING Sector repair completed: port=1, LBA=0x323C1AC<br />
c0 [Sat May 26 00:51:43 2012] WARNING Sector repair completed: port=1, LBA=0x323C1AE<br />
c0 [Sat May 26 01:54:21 2012] WARNING Sector repair completed: port=1, LBA=0x2F0D302<br />
c0 [Sat May 26 02:06:38 2012] WARNING Sector repair completed: port=0, LBA=0x12DA777<br />
c0 [Sat May 26 02:07:21 2012] WARNING Sector repair completed: port=0, LBA=0x12E48FE<br />
c0 [Sat May 26 04:20:00 2012] WARNING Sector repair completed: port=1, LBA=0x2F0D306<br />
c0 [Sat May 26 04:32:58 2012] WARNING Sector repair completed: port=1, LBA=0x323C1B1<br />
c0 [Sat May 26 04:33:21 2012] WARNING Sector repair completed: port=1, LBA=0x323C1B3<br />
c0 [Sat May 26 04:33:44 2012] WARNING Sector repair completed: port=1, LBA=0x323C1BA<br />
c0 [Sat May 26 05:24:07 2012] WARNING Sector repair completed: port=1, LBA=0x3F83862<br />
c0 [Sat May 26 05:25:09 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 06:08:13 2012] WARNING Sector repair completed: port=0, LBA=0x4CDC6A2<br />
c0 [Sat May 26 09:49:35 2012] WARNING Sector repair completed: port=1, LBA=0x6CACD4A<br />
c0 [Sat May 26 18:10:44 2012] WARNING Sector repair completed: port=1, LBA=0x18F425EA<br />
c0 [Sat May 26 19:45:40 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:22:52 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:15 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:22 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:35 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:41 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:49 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:57 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:24:02 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:54:41 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 22:00:30 2012] INFO Verify completed: unit=0<br />
c0 [Sat Jun 2 00:01:43 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 2 00:30:17 2012] WARNING Sector repair completed: port=0, LBA=0x2B911E4<br />
c0 [Sat Jun 2 00:50:57 2012] WARNING Sector repair completed: port=0, LBA=0x5A807CA6<br />
c0 [Sat Jun 2 04:13:13 2012] WARNING Sector repair completed: port=0, LBA=0x2D18291<br />
c0 [Sat Jun 2 04:13:35 2012] WARNING Sector repair completed: port=0, LBA=0x2D1829F<br />
c0 [Sat Jun 2 21:48:02 2012] INFO Verify completed: unit=0<br />
c0 [Mon Jun 4 04:40:34 2012] WARNING Sector repair completed: port=1, LBA=0x4AF8098F<br />
c0 [Tue Jun 5 00:28:19 2012] WARNING Sector repair completed: port=1, LBA=0x263C5CD<br />
c0 [Tue Jun 5 00:33:06 2012] WARNING Sector repair completed: port=1, LBA=0x263C5CF<br />
c0 [Thu Jun 7 00:34:27 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B5F<br />
c0 [Thu Jun 7 00:38:50 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B61<br />
c0 [Fri Jun 8 00:07:13 2012] WARNING Sector repair completed: port=0, LBA=0xC131F6B<br />
c0 [Sat Jun 9 00:01:41 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 9 00:29:11 2012] WARNING Sector repair completed: port=0, LBA=0x6C7614D<br />
c0 [Sat Jun 9 00:38:25 2012] WARNING Sector repair completed: port=0, LBA=0x6C76152<br />
c0 [Sat Jun 9 04:02:30 2012] WARNING Sector repair completed: port=1, LBA=0x263C5D1<br />
c0 [Sat Jun 9 04:02:52 2012] WARNING Sector repair completed: port=1, LBA=0x263C5D3<br />
c0 [Sat Jun 9 04:07:32 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E12<br />
c0 [Sat Jun 9 04:07:57 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E15<br />
c0 [Sat Jun 9 04:08:16 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E17<br />
c0 [Sat Jun 9 04:08:45 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E19<br />
c0 [Sat Jun 9 04:15:04 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B64<br />
c0 [Sat Jun 9 04:15:26 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B66<br />
c0 [Sat Jun 9 04:15:45 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B68<br />
c0 [Sat Jun 9 04:15:59 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B6C<br />
c0 [Sat Jun 9 04:16:13 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B6E<br />
c0 [Sat Jun 9 21:48:52 2012] INFO Verify completed: unit=0<br />
c0 [Thu Jun 14 00:40:10 2012] WARNING Sector repair completed: port=0, LBA=0x334F14B<br />
c0 [Sat Jun 16 00:01:38 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 16 21:16:19 2012] INFO Verify completed: unit=0<br />
c0 [Tue Jun 19 02:03:43 2012] WARNING Sector repair completed: port=1, LBA=0xFE41EAD<br />
c0 [Wed Jun 20 02:30:02 2012] WARNING Sector repair completed: port=1, LBA=0xD99145C<br />
c0 [Sat Jun 23 00:01:36 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 23 04:27:04 2012] WARNING Sector repair completed: port=1, LBA=0x2FAD311<br />
c0 [Sat Jun 23 06:52:38 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC8D<br />
c0 [Sat Jun 23 06:53:03 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC91<br />
c0 [Sat Jun 23 06:53:21 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC94<br />
c0 [Sat Jun 23 17:00:22 2012] WARNING Sector repair completed: port=1, LBA=0xF9AC7C9<br />
c0 [Sat Jun 23 21:15:19 2012] INFO Verify completed: unit=0<br />
c0 [Sat Jun 30 00:01:34 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 30 05:24:13 2012] WARNING Sector repair completed: port=0, LBA=0x3FAA9E7<br />
c0 [Sat Jun 30 14:49:39 2012] WARNING Sector repair completed: port=1, LBA=0x869931C<br />
c0 [Sat Jun 30 21:31:05 2012] INFO Verify completed: unit=0<br />
c0 [Tue Jul 3 03:40:25 2012] WARNING Sector repair completed: port=1, LBA=0xD36C7F7<br />
c0 [Fri Jul 6 02:50:18 2012] WARNING Sector repair completed: port=1, LBA=0x3562470<br />
c0 [Fri Jul 6 22:18:26 2012] WARNING Sector repair completed: port=1, LBA=0x3563173<br />
c0 [Sat Jul 7 00:01:31 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 7 00:50:16 2012] WARNING Sector repair completed: port=0, LBA=0x76EE88<br />
c0 [Sat Jul 7 00:50:39 2012] WARNING Sector repair completed: port=0, LBA=0x76EE8F<br />
c0 [Sat Jul 7 21:39:36 2012] INFO Verify completed: unit=0<br />
c0 [Sun Jul 8 02:51:05 2012] WARNING Sector repair completed: port=0, LBA=0x67759D<br />
c0 [Sun Jul 8 02:53:55 2012] WARNING Sector repair completed: port=0, LBA=0x67759B<br />
c0 [Tue Jul 10 16:17:21 2012] WARNING Sector repair completed: port=0, LBA=0x15C8C695<br />
c0 [Wed Jul 11 22:51:22 2012] WARNING Sector repair completed: port=1, LBA=0x355BBD0<br />
c0 [Sat Jul 14 00:01:28 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 14 01:33:40 2012] WARNING Sector repair completed: port=1, LBA=0x1333BCF4<br />
c0 [Sat Jul 14 03:36:23 2012] WARNING Sector repair completed: port=1, LBA=0x2174773<br />
c0 [Sat Jul 14 11:26:44 2012] WARNING Sector repair completed: port=0, LBA=0x7429AB7<br />
c0 [Sat Jul 14 16:53:50 2012] WARNING Sector repair completed: port=1, LBA=0xA17EB3F<br />
c0 [Sat Jul 14 21:19:25 2012] INFO Verify completed: unit=0<br />
c0 [Wed Jul 18 05:08:47 2012] WARNING Sector repair completed: port=1, LBA=0x17D62EDC<br />
c0 [Wed Jul 18 05:14:15 2012] WARNING Sector repair completed: port=1, LBA=0x17D62EE1<br />
c0 [Thu Jul 19 03:24:59 2012] WARNING Sector repair completed: port=0, LBA=0x7733C3D<br />
c0 [Thu Jul 19 03:25:20 2012] WARNING Sector repair completed: port=0, LBA=0x773CEA5<br />
c0 [Thu Jul 19 03:28:16 2012] WARNING Sector repair completed: port=0, LBA=0x7733C42<br />
c0 [Thu Jul 19 03:28:41 2012] WARNING Sector repair completed: port=0, LBA=0x773CEAF<br />
c0 [Sat Jul 21 00:01:26 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 21 03:07:30 2012] WARNING Sector repair completed: port=1, LBA=0x1CC6936<br />
c0 [Sat Jul 21 03:07:52 2012] WARNING Sector repair completed: port=1, LBA=0x1CC6938<br />
c0 [Sat Jul 21 03:08:11 2012] WARNING Sector repair completed: port=1, LBA=0x1CC693A<br />
c0 [Sat Jul 21 16:43:56 2012] WARNING Sector repair completed: port=0, LBA=0xD04C914<br />
c0 [Sat Jul 21 16:45:31 2012] WARNING Sector repair completed: port=1, LBA=0xD456973<br />
c0 [Sat Jul 21 21:14:29 2012] INFO Verify completed: unit=0<br />
c0 [Wed Jul 25 03:37:25 2012] WARNING Sector repair completed: port=0, LBA=0x1F8E6C43<br />
c0 [Sat Jul 28 00:01:24 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 28 01:45:27 2012] WARNING Sector repair completed: port=0, LBA=0x11584AD<br />
c0 [Sat Jul 28 18:54:25 2012] WARNING Sector repair completed: port=1, LBA=0x447C3E6C<br />
c0 [Sat Jul 28 21:13:46 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 1 03:20:11 2012] WARNING Sector repair completed: port=0, LBA=0x805FEF<br />
c0 [Fri Aug 3 00:50:03 2012] WARNING Sector repair completed: port=0, LBA=0xCED0ACA<br />
c0 [Sat Aug 4 00:01:22 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 4 00:52:51 2012] WARNING Sector repair completed: port=0, LBA=0x805FF3<br />
c0 [Sat Aug 4 00:53:14 2012] WARNING Sector repair completed: port=0, LBA=0x805FF5<br />
c0 [Sat Aug 4 00:53:33 2012] WARNING Sector repair completed: port=0, LBA=0x805FF7<br />
c0 [Sat Aug 4 00:53:47 2012] WARNING Sector repair completed: port=0, LBA=0x805FF9<br />
c0 [Sat Aug 4 00:54:00 2012] WARNING Sector repair completed: port=0, LBA=0x805FFB<br />
c0 [Sat Aug 4 00:54:14 2012] WARNING Sector repair completed: port=0, LBA=0x805FFD<br />
c0 [Sat Aug 4 00:54:27 2012] WARNING Sector repair completed: port=0, LBA=0x805FFF<br />
c0 [Sat Aug 4 04:43:12 2012] WARNING Sector repair completed: port=1, LBA=0x16974289<br />
c0 [Sat Aug 4 04:58:17 2012] WARNING Sector repair completed: port=1, LBA=0x1697428E<br />
c0 [Sat Aug 4 20:54:53 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 8 03:21:55 2012] ERROR Drive timeout detected: port=1<br />
c0 [Wed Aug 8 15:31:44 2012] WARNING Sector repair completed: port=0, LBA=0x1A366CD3<br />
c0 [Sat Aug 11 00:01:21 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 11 20:40:51 2012] INFO Verify completed: unit=0<br />
c0 [Thu Aug 16 05:10:55 2012] WARNING Sector repair completed: port=0, LBA=0x1C22593<br />
c0 [Sat Aug 18 00:01:18 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 18 03:00:20 2012] WARNING Sector repair completed: port=0, LBA=0x1C225A5<br />
c0 [Sat Aug 18 03:43:00 2012] WARNING Sector repair completed: port=1, LBA=0x23EE91E<br />
c0 [Sat Aug 18 03:43:23 2012] WARNING Sector repair completed: port=1, LBA=0x23EE920<br />
c0 [Sat Aug 18 17:00:06 2012] WARNING Sector repair completed: port=1, LBA=0x137D066A<br />
c0 [Sat Aug 18 17:00:29 2012] WARNING Sector repair completed: port=1, LBA=0x137D066D<br />
c0 [Sat Aug 18 21:13:01 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 22 01:36:08 2012] WARNING Sector repair completed: port=0, LBA=0x2560A0F<br />
c0 [Wed Aug 22 01:37:42 2012] WARNING Sector repair completed: port=0, LBA=0x2560A13<br />
c0 [Fri Aug 24 04:01:36 2012] WARNING Sector repair completed: port=1, LBA=0x55C1A5DF<br />
c0 [Fri Aug 24 05:02:06 2012] WARNING Sector repair completed: port=1, LBA=0xCE3378A<br />
c0 [Sat Aug 25 00:01:17 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 25 00:31:06 2012] WARNING Sector repair completed: port=1, LBA=0x50F65D<br />
c0 [Sat Aug 25 00:39:52 2012] WARNING Sector repair completed: port=0, LBA=0x678FF4<br />
c0 [Sat Aug 25 03:43:15 2012] WARNING Sector repair completed: port=0, LBA=0x2560A15<br />
c0 [Sat Aug 25 03:43:39 2012] WARNING Sector repair completed: port=0, LBA=0x2560A19<br />
c0 [Sat Aug 25 03:43:58 2012] WARNING Sector repair completed: port=0, LBA=0x2560A1B<br />
c0 [Sat Aug 25 03:44:30 2012] WARNING Sector repair completed: port=0, LBA=0x2560A21<br />
c0 [Sat Aug 25 20:58:14 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 29 04:57:15 2012] WARNING Sector repair completed: port=1, LBA=0xF3957EB<br />
c0 [Sat Sep 1 00:01:15 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 1 03:21:52 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC86<br />
c0 [Sat Sep 1 03:22:15 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC88<br />
c0 [Sat Sep 1 03:22:34 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC8A<br />
c0 [Sat Sep 1 03:22:47 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC8C<br />
c0 [Sat Sep 1 17:17:22 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD1<br />
c0 [Sat Sep 1 17:17:45 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD3<br />
c0 [Sat Sep 1 17:18:04 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD5<br />
c0 [Sat Sep 1 21:36:56 2012] INFO Verify completed: unit=0<br />
c0 [Thu Sep 6 00:07:30 2012] WARNING Sector repair completed: port=0, LBA=0xDA3C64B<br />
c0 [Thu Sep 6 00:32:56 2012] WARNING Sector repair completed: port=1, LBA=0x6BBA816<br />
c0 [Sat Sep 8 00:01:13 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 8 00:09:56 2012] WARNING Sector repair completed: port=0, LBA=0xDEBC958<br />
c0 [Sat Sep 8 04:38:45 2012] WARNING Sector repair completed: port=0, LBA=0x38D254F<br />
c0 [Sat Sep 8 20:44:50 2012] INFO Verify completed: unit=0<br />
c0 [Mon Sep 10 01:26:34 2012] WARNING Sector repair completed: port=1, LBA=0xFFD8D5E<br />
c0 [Wed Sep 12 00:33:48 2012] WARNING Sector repair completed: port=1, LBA=0xE8DB928<br />
c0 [Wed Sep 12 00:36:33 2012] WARNING Sector repair completed: port=1, LBA=0x6D49411<br />
c0 [Fri Sep 14 01:59:39 2012] WARNING Sector repair completed: port=0, LBA=0x1467F1C<br />
c0 [Fri Sep 14 02:08:27 2012] WARNING Sector repair completed: port=1, LBA=0x14C8ABD<br />
c0 [Fri Sep 14 03:54:47 2012] WARNING Sector repair completed: port=0, LBA=0x1580C915<br />
c0 [Sat Sep 15 00:01:11 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 15 02:38:14 2012] WARNING Sector repair completed: port=0, LBA=0x1C178973<br />
c0 [Sat Sep 15 02:59:02 2012] WARNING Sector repair completed: port=0, LBA=0x1C178975<br />
c0 [Sat Sep 15 04:47:08 2012] WARNING Sector repair completed: port=0, LBA=0x3FA0356<br />
c0 [Sat Sep 15 04:47:31 2012] WARNING Sector repair completed: port=0, LBA=0x3FA0359<br />
c0 [Sat Sep 15 10:41:59 2012] WARNING Sector repair completed: port=0, LBA=0x6DFD1EC<br />
c0 [Sat Sep 15 13:25:23 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD100<br />
c0 [Sat Sep 15 13:25:31 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD104<br />
c0 [Sat Sep 15 13:25:54 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD106<br />
c0 [Sat Sep 15 17:10:50 2012] WARNING Sector repair completed: port=0, LBA=0x1C178977<br />
c0 [Sat Sep 15 20:59:57 2012] INFO Verify completed: unit=0<br />
c0 [Tue Sep 18 01:17:18 2012] WARNING Sector repair completed: port=1, LBA=0x803B05B<br />
c0 [Sat Sep 22 00:01:10 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 22 20:54:31 2012] INFO Verify completed: unit=0<br />
c0 [Tue Sep 25 01:56:47 2012] WARNING Sector repair completed: port=0, LBA=0x26E3909<br />
c0 [Sat Sep 29 00:01:08 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 29 02:04:14 2012] WARNING Sector repair completed: port=0, LBA=0x146AC03<br />
c0 [Sat Sep 29 10:58:39 2012] WARNING Sector repair completed: port=0, LBA=0x6D4EB0E<br />
c0 [Sat Sep 29 10:59:02 2012] WARNING Sector repair completed: port=0, LBA=0x6D4EB14<br />
c0 [Sat Sep 29 11:22:44 2012] WARNING Sector repair completed: port=0, LBA=0x6F79623<br />
c0 [Sat Sep 29 13:50:48 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D65E<br />
c0 [Sat Sep 29 13:51:11 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D661<br />
c0 [Sat Sep 29 13:51:30 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D663<br />
c0 [Sat Sep 29 20:57:34 2012] INFO Verify completed: unit=0<br />
c0 [Mon Oct 1 04:47:24 2012] WARNING Sector repair completed: port=0, LBA=0xC5BC6F2<br />
c0 [Tue Oct 2 02:00:27 2012] WARNING Sector repair completed: port=0, LBA=0x1547667<br />
c0 [Tue Oct 2 02:01:56 2012] WARNING Sector repair completed: port=0, LBA=0x154766F<br />
c0 [Tue Oct 2 05:02:31 2012] WARNING Sector repair completed: port=1, LBA=0xD67D054<br />
c0 [Tue Oct 2 05:04:14 2012] WARNING Sector repair completed: port=1, LBA=0xD67D056<br />
c0 [Wed Oct 3 01:22:12 2012] WARNING Sector repair completed: port=1, LBA=0x12AAF8CA<br />
c0 [Thu Oct 4 04:29:22 2012] WARNING Sector repair completed: port=0, LBA=0x13E6F992<br />
c0 [Thu Oct 4 05:10:51 2012] WARNING Sector repair completed: port=0, LBA=0x1C252A4<br />
c0 [Sat Oct 6 00:01:07 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 6 19:41:18 2012] WARNING Sector repair completed: port=1, LBA=0x5A5C3AE8<br />
c0 [Sat Oct 6 21:01:05 2012] INFO Verify completed: unit=0<br />
c0 [Mon Oct 8 00:32:06 2012] WARNING Sector repair completed: port=0, LBA=0x6C60D3E<br />
c0 [Tue Oct 9 03:51:03 2012] WARNING Sector repair completed: port=1, LBA=0x89B5EC9<br />
c0 [Thu Oct 11 04:21:17 2012] WARNING Sector repair completed: port=1, LBA=0x13F85833<br />
c0 [Sat Oct 13 00:01:05 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 13 05:12:40 2012] WARNING Sector repair completed: port=0, LBA=0x3FA5134<br />
c0 [Sat Oct 13 21:08:35 2012] INFO Verify completed: unit=0<br />
c0 [Tue Oct 16 03:53:50 2012] WARNING Sector repair completed: port=1, LBA=0x148AA1BD<br />
c0 [Thu Oct 18 03:20:30 2012] WARNING Sector repair completed: port=0, LBA=0x1C8DABCB<br />
c0 [Thu Oct 18 04:52:50 2012] WARNING Sector repair completed: port=0, LBA=0xE879057<br />
c0 [Sat Oct 20 00:01:04 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 20 02:19:25 2012] WARNING Sector repair completed: port=1, LBA=0x174B012<br />
c0 [Sat Oct 20 03:41:38 2012] WARNING Sector repair completed: port=0, LBA=0x256D93B<br />
c0 [Sat Oct 20 03:42:01 2012] WARNING Sector repair completed: port=0, LBA=0x256D93D<br />
c0 [Sat Oct 20 03:42:40 2012] WARNING Sector repair completed: port=0, LBA=0x256D940<br />
c0 [Sat Oct 20 03:42:59 2012] WARNING Sector repair completed: port=0, LBA=0x256D942<br />
c0 [Sat Oct 20 03:43:12 2012] WARNING Sector repair completed: port=0, LBA=0x256D944<br />
c0 [Sat Oct 20 03:43:26 2012] WARNING Sector repair completed: port=0, LBA=0x256D948<br />
c0 [Sat Oct 20 16:37:52 2012] WARNING Sector repair completed: port=0, LBA=0xE879060<br />
c0 [Sat Oct 20 16:38:15 2012] WARNING Sector repair completed: port=0, LBA=0xE879062<br />
c0 [Sat Oct 20 21:00:18 2012] INFO Verify completed: unit=0<br />
c0 [Sat Oct 20 23:49:01 2012] WARNING Sector repair completed: port=1, LBA=0x4473E908<br />
c0 [Sun Oct 21 03:42:26 2012] WARNING Sector repair completed: port=1, LBA=0x175BADD5<br />
c0 [Tue Oct 23 01:09:04 2012] WARNING Sector repair completed: port=1, LBA=0x6E524860<br />
c0 [Fri Oct 26 03:21:25 2012] WARNING Sector repair completed: port=0, LBA=0x802C61<br />
c0 [Fri Oct 26 04:22:21 2012] WARNING Sector repair completed: port=0, LBA=0x176353CD<br />
c0 [Sat Oct 27 00:01:03 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 27 00:49:35 2012] WARNING Sector repair completed: port=0, LBA=0x802C65<br />
c0 [Sat Oct 27 17:02:24 2012] WARNING Sector repair completed: port=1, LBA=0xC1FF26D<br />
c0 [Sat Oct 27 17:09:06 2012] WARNING Sector repair completed: port=0, LBA=0xDF621AD<br />
c0 [Sat Oct 27 21:30:57 2012] INFO Verify completed: unit=0<br />
c0 [Tue Oct 30 00:20:46 2012] WARNING Sector repair completed: port=0, LBA=0xE9FE2AB<br />
c0 [Wed Oct 31 02:02:03 2012] WARNING Sector repair completed: port=0, LBA=0x1460C25<br />
c0 [Wed Oct 31 02:04:05 2012] WARNING Sector repair completed: port=0, LBA=0x1460C28<br />
c0 [Thu Nov 1 00:48:34 2012] WARNING Sector repair completed: port=1, LBA=0xA7C92BE<br />
c0 [Thu Nov 1 05:04:45 2012] WARNING Sector repair completed: port=0, LBA=0x1C252C2<br />
<br />
[root@backup1 /data/deprecated]#<br />
</pre><br />
<br />
Look for failed drives and errors. Obviously from the above we need to probably replace drives 0 and 1 and drive 1 is even showing as having problems, yet the RAID array is healthy, amazingly. You also see the automatic verifies.<br />
<br />
Note: when rebuilding a degraded mirror, you will see no progress as it rebuilds in the cli<br />
<pre>Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u0 RAID-5 REBUILDING 0 - 64K 4656.56 OFF ON</pre><br />
<br />
<br />
=== areca ===<br />
<br />
We are using an areca controller on backup3. <br />
<br />
<pre><br />
[root@newbackup3 ~]# sh /root/verify.sh<br />
# Name Raid Name Level Capacity Ch/Id/Lun State<br />
===============================================================================<br />
1 ARC-1160-VOL#00 Raid Set # 00 Raid5 5000.0GB 00/00/00 Checking(19.7%)<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
# Name Disks TotalCap FreeCap DiskChannels State<br />
===============================================================================<br />
1 Raid Set # 00 6 6000.0GB 0.0GB 123456 Checking<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
Date-Time Device Event Type Elapsed Time Errors<br />
===============================================================================<br />
2012-12-05 20:40:58 ARC-1160-VOL#00 Start Checking<br />
2012-12-01 05:06:04 ARC-1160-VOL#00 Complete Init 027:30:45<br />
2012-11-30 01:35:19 ARC-1160-VOL#00 Start Initialize<br />
2026-08-06 01:34:52 H/W MONITOR Raid Powered On<br />
2012-11-30 01:33:36 ARC-1160-VOL#00 Stop Initialization 000:31:48<br />
2012-11-30 01:01:47 ARC-1160-VOL#00 Start Initialize<br />
2026-08-06 00:58:13 H/W MONITOR Raid Powered On<br />
2012-11-30 00:57:26 ARC-1160-VOL#00 Stop Initialization 000:57:07<br />
2012-11-30 00:00:19 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 23:56:48 H/W MONITOR Raid Powered On<br />
2026-08-05 23:52:58 H/W MONITOR Raid Powered On<br />
2026-08-05 23:50:14 H/W MONITOR Raid Powered On<br />
2026-08-05 23:43:30 H/W MONITOR Raid Powered On<br />
2012-11-29 23:10:07 ARC-1160-VOL#00 Stop Initialization 000:00:56<br />
2012-11-29 23:09:11 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 23:08:57 H/W MONITOR Raid Powered On<br />
2012-11-29 23:08:10 ARC-1160-VOL#00 Stop Initialization 000:20:41<br />
2012-11-29 22:47:29 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 22:46:59 H/W MONITOR Raid Powered On<br />
2026-08-05 22:45:55 H/W MONITOR Raid Powered On<br />
2026-08-05 22:44:53 H/W MONITOR Raid Powered On<br />
2026-08-05 22:42:06 H/W MONITOR Raid Powered On<br />
2026-08-05 22:40:50 H/W MONITOR Raid Powered On<br />
2012-11-29 22:40:04 ARC-1160-VOL#00 Stop Initialization 000:24:25<br />
2012-11-29 22:15:38 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 22:15:11 000:000001215B00 Restart Init LBA Point<br />
2026-08-05 22:15:10 H/W MONITOR Raid Powered On<br />
2012-11-29 21:56:38 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 21:56:12 H/W MONITOR Raid Powered On<br />
2026-08-05 21:56:04 IDE Channel #03 Device Inserted<br />
2012-11-29 21:55:13 IDE Channel #04 Device Inserted<br />
2012-11-29 21:55:03 IDE Channel #02 Device Inserted<br />
2026-08-05 21:53:09 H/W MONITOR Raid Powered On<br />
2026-08-05 20:51:46 H/W MONITOR Raid Powered On<br />
2026-08-05 20:49:56 H/W MONITOR Raid Powered On<br />
2026-08-05 20:48:29 H/W MONITOR Raid Powered On<br />
2026-08-05 20:46:29 H/W MONITOR Raid Powered On<br />
2026-08-05 20:44:49 H/W MONITOR Raid Powered On<br />
2026-08-05 20:43:01 H/W MONITOR Raid Powered On<br />
2026-08-05 20:36:25 H/W MONITOR Raid Powered On<br />
2026-08-05 20:31:18 H/W MONITOR Raid Powered On<br />
2026-08-05 20:30:08 H/W MONITOR Raid Powered On<br />
2026-08-05 20:08:40 H/W MONITOR Raid Powered On<br />
2026-08-05 20:06:11 H/W MONITOR Raid Powered On<br />
2026-08-05 20:05:14 H/W MONITOR Raid Powered On<br />
2026-08-05 20:03:58 H/W MONITOR Raid Powered On<br />
2026-08-05 20:00:56 H/W MONITOR Raid Powered On<br />
2026-08-05 19:57:57 H/W MONITOR Raid Powered On<br />
2026-08-05 19:56:15 H/W MONITOR Raid Powered On<br />
2026-08-05 19:55:05 H/W MONITOR Raid Powered On<br />
2026-08-05 17:24:36 H/W MONITOR Raid Powered On<br />
2026-08-05 17:22:43 H/W MONITOR Raid Powered On<br />
2026-08-05 04:50:42 H/W MONITOR Raid Powered On<br />
2026-08-05 04:47:33 H/W MONITOR Raid Powered On<br />
2026-08-05 04:43:57 H/W MONITOR Raid Powered On<br />
2026-08-05 04:18:52 H/W MONITOR Raid Powered On<br />
2026-08-05 04:17:30 H/W MONITOR Raid Powered On<br />
2026-08-05 04:13:30 H/W MONITOR Raid Powered On<br />
2026-08-05 04:10:26 H/W MONITOR Raid Powered On<br />
2026-08-05 04:09:23 H/W MONITOR Raid Powered On<br />
2026-08-05 00:08:09 H/W MONITOR Raid Powered On<br />
2026-08-05 00:07:12 H/W MONITOR Raid Powered On<br />
2026-08-05 00:05:51 H/W MONITOR Raid Powered On<br />
2026-08-05 00:04:27 H/W MONITOR Raid Powered On<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
press enter when ready to run verify<br />
</pre><br />
<br />
Look for failed drives and errors. <br />
<br />
When it proceed's to verifying, you can confirm with:<br />
<br />
<pre><br />
[root@newbackup3 ~]# cli64 vsf info<br />
# Name Raid Name Level Capacity Ch/Id/Lun State<br />
===============================================================================<br />
1 ARC-1160-VOL#00 Raid Set # 00 Raid5 5000.0GB 00/00/00 Checking(22.5%)<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
[root@newbackup3 ~]#<br />
</pre><br />
<br />
== Update OS list ==<br />
<br />
# check for any new VZ templates we want to offer: <tt>vzup2date -z</tt><br />
# see if there's any OS's we want to include in our colo install list. Update 2 places: <tt>signup/html/colo_quote.html</tt> & <tt>signup/html/step1.html</tt><br />
# update the mgmt database (ref_templates table, ref_systems table).<br />
<br />
= Infrequent tasks =<br />
<br />
== Free up space on gateway ==<br />
<br />
<pre>newgateway /var/spool# cd clientmqueue/<br />
newgateway /var/spool/clientmqueue# sh<br />
# for f in `ls`; do rm $f; done<br />
exit</pre><br />
<br />
== Free up space on mail ==<br />
<br />
You can clear out root mail:<br />
<pre><br />
mail /var/log# ll -h /var/mail/root<br />
-rw------- 1 root mail 543K Dec 19 13:05 /var/mail/root<br />
mail /var/log# rm /var/mail/root<br />
</pre><br />
<br />
Or you can archive mail logs:<br />
<pre><br />
mail /var/log# ls -l htt*<br />
-rw-r--r-- 1 root wheel 297436931 Dec 19 13:26 httpd-access.log<br />
-rw-r--r-- 1 root wheel 9824324 Jul 4 11:34 httpd-access.log.old.0.gz<br />
-rw-r--r-- 1 root wheel 6884137 Mar 17 2012 httpd-access.log.old.1.gz<br />
-rw-r--r-- 1 root wheel 18557444 Dec 3 2009 httpd-access.log.old.10.gz<br />
-rw-r--r-- 1 root wheel 14740263 Jan 9 2007 httpd-access.log.old.11.gz<br />
-rw-r--r-- 1 root wheel 14209465 Nov 28 2007 httpd-access.log.old.12.gz<br />
-rw-r--r-- 1 root wheel 16874396 Feb 19 2012 httpd-access.log.old.3.gz<br />
-rw-r--r-- 1 root wheel 14554859 Jul 22 2011 httpd-access.log.old.4.gz<br />
-rw-r--r-- 1 root wheel 10513227 Feb 18 2011 httpd-access.log.old.5.gz<br />
-rw-r--r-- 1 root wheel 7201946 Oct 29 2010 httpd-access.log.old.6.gz<br />
-rw-r--r-- 1 root wheel 10062537 May 6 2010 httpd-access.log.old.7.gz<br />
-rw-r--r-- 1 root wheel 10157042 Aug 12 2010 httpd-access.log.old.8.gz<br />
-rw-r--r-- 1 root wheel 11909534 Mar 4 2010 httpd-access.log.old.9.gz<br />
-rw-r--r-- 1 root wheel 59030930 Dec 19 13:01 httpd-error.log<br />
-rw-r--r-- 1 root wheel 3413134 Mar 4 2010 httpd-error.log.0.gz<br />
-rw-r--r-- 1 root wheel 795515 May 1 2007 httpd-error.log.1.gz<br />
-rw-r--r-- 1 root wheel 1142153 Nov 30 2007 httpd-error.log.2.gz<br />
-rw-r--r-- 1 root wheel 2325801 Feb 18 2011 httpd-error.log.gz<br />
<br />
mail /var/log# sh<br />
# for f in 12 11 10 9 8 7 6 5 4 3 2 1 0; do g=`echo $f+1|bc`; mv httpd-access.log.old.$f.gz httpd-access.log.old.$g.gz; done<br />
# mv httpd-access.log httpd-access.log.old.0<br />
# touch httpd-access.log<br />
# apachectl restart<br />
# gzip httpd-access.log.old.0<br />
<br />
# for f in 2 1 0; do g=`echo $f+1|bc`; mv httpd-error.log.$f.gz httpd-error.log.$g.gz; done<br />
# mv httpd-error.log httpd-error.log.0<br />
# touch httpd-error.log<br />
# apachectl restart<br />
# gzip httpd-error.log.0<br />
# exit<br />
</pre><br />
<br />
== Free up space on bwdb2 ==<br />
<br />
You can either remove items from <tt>/usr/home/archive</tt> or you can scp them to backup3:/data/bwdb2/archive .<br />
<br />
== Free up space on backup1 ==<br />
<br />
backup1 is our primary customer backup system. As usage grows over time, it needs to be regularly purged of old files. The easiest way to do this is by removing deprecated files. These mostly consist of cancelled customers or temporary dump/storage files (created during dump/restores). Our standard policy is to hang onto cancelled customers for 6mos after which we remove their files (as far as customers know their data is purged immediately, but we hang onto it just in case.. and in some cases we cancel a server due to non payment so this makes it easy to restore their system). To find files to remove:<br />
<br />
<pre>[root@backup1 ~]# cd /data/deprecated/<br />
[root@backup1 /data/deprecated]# ls<br />
2101-migrated-20120317.tgz old-683-cxld-20121021.tgz<br />
69.55.230.2-wwwbackup old-744-cxld-20120708.tgz<br />
991-DONTDELETE.tgz old-809-cxld-20120609.tgz<br />
archive-col02050-mdfile-cxld-20120409.gz old-854-cxld-20120621.tgz<br />
col01371.tgz old-931-cxld-20060513.tgz<br />
deleteme_ubuntu-10.10-x86_20111205 old-col00123-mdfile-noarchive-20120417.gz<br />
jail10_old old-col00147-vnfile-cxld-20120828.gz<br />
jail14_rsync_old old-col00419-dump-cxld-20120224.gz<br />
jail15_old old-col01098-vnfile-cxld-20120827.gz<br />
jail3_old old-col01278-dump-cxld-20120822<br />
jail4_old old-col01517-dump-cxld-20120828<br />
jail5_old old-col01669-dump-cxld-20120203.gz<br />
old-1009-cxld-20120608.tgz old-col01687-dump-cxld-20120909<br />
old-1012-cxld-20120411.tgz old-col01790-dump-cxld-20120828<br />
old-1052-cxld-20120721.tgz old-col01812-dump-cxld-20120820<br />
old-10631-cxld-20120622.tgz old-col01938-mdfile-cxld-20120619.gz<br />
old-10632-cxld-20120622.tgz old-col02095-mdfile-noarchive-20120523.gz<br />
old-10633-cxld-20120622.tgz olddebian-3.0-v15-20110610.tgz<br />
old-1236-cxld-20120621.tgz oldmod_frontpage-deb30-v15-20110610.tgz<br />
old-1381-cxld-20120404.tgz oldmod_perl-deb30-v15-20110610.tgz<br />
old-1422-cxld-20120721.tgz oldmod_ssl-deb30-v15-20110610.tgz<br />
old-14681-cxld-20120619.tgz oldmysql-deb30-v15-20110610.tgz<br />
old-1544-cxld-20120626.tgz oldproftpd-deb30-v15-20110610.tgz<br />
old-18351-cxld-20120605.tgz old_virt14<br />
old-1853-cxld-20120910.tgz old_virt18<br />
old-1963-cxld-20120206.tgz oldwebmin-deb30-v15-20110610.tgz<br />
old-1967-cxld-20120605.tgz suse.virt11.20120421.tgz<br />
old-1981-noarchive-20120729.tgz virt11<br />
old-2030-migrated-noarchive-20120727.tgz virt12_old<br />
old-2037-cxld-20120716.tgz virt13_old<br />
old-2065-cxld-20120727.tgz virt16_old<br />
old-2068-cxld-20120424.tgz virt4_old<br />
old-2085-cxld-20120531.tgz virt5_old<br />
old-364-cxld-20120904.tgz virt6_old<br />
old-446-cxld-20120512.tgz virt7_old<br />
old-613-cxld-20120601.tgz virt8_old<br />
[root@backup1 /data/deprecated]#<br />
</pre><br />
<br />
virtX_old and jailX_old are permanently archived, so ignore those as well as anything else marked not to delete or otherwise suspicious. Likewise, probably a good idea to try to hang onto oldTEMPLATE.gz as long as we can as well.<br />
Most of the stuff we want to delete is dated when it was deprecated, making this easy. So to remove files from 6 mos ago (running this in Oct):<br />
<br />
<pre>[root@backup1 /data/deprecated]# ls old*201204*<br />
old-1012-cxld-20120411.tgz old-2068-cxld-20120424.tgz<br />
old-1381-cxld-20120404.tgz old-col00123-mdfile-noarchive-20120417.gz<br />
[root@backup1 /data/deprecated]# rm old*201204*<br />
</pre><br />
<br />
Every few months you will also want to remove some of the snapshot archives for mail. We typically save the 1st, 10th, and 20th of each month. To do this you set aside the dates you want to save then remove months at a time, followed by restoring the set aside dates. Here's how that works:<br />
<pre>[root@backup1 /data/www/daily]# ls<br />
05 08-10-11 10-04-10 11-10-10 12-07-29 12-09-21 12-11-14<br />
06 08-10-21 10-04-20 11-10-20 12-07-30 12-09-22 12-11-15<br />
06-06-01-usr-home.tgz 08-11-01 10-05-01 11-11-01 12-07-31 12-09-23 12-11-16<br />
06-07-01-usr-home.tgz 08-11-10 10-05-11 11-11-10 12-08-01 12-09-24 12-11-17<br />
06-08-01-usr-home.tgz 08-11-20 10-05-20 11-11-20 12-08-02 12-09-25 12-11-18<br />
06-09-01-usr-home.tgz 08-12-01 10-06-01 11-12-01 12-08-03 12-09-26 12-11-19<br />
06-11-10 08-12-10 10-06-10 11-12-10 12-08-04 12-09-27 12-11-20<br />
06-12-21 08-12-20 10-06-20 11-12-20 12-08-05 12-09-28 12-11-21<br />
07-01-10 09-01-01 10-07-01 12-01-01 12-08-06 12-09-29 12-11-22<br />
07-01-20 09-01-10 10-07-10 12-01-10 12-08-07 12-09-30 12-11-23<br />
07-02-10 09-01-20 10-07-20 12-01-20 12-08-08 12-10-01 12-11-24<br />
07-02-20 09-02-01 10-08-01 12-02-01 12-08-09 12-10-02 12-11-25<br />
07-03-01 09-02-10 10-08-10 12-02-10 12-08-10 12-10-03 12-11-26<br />
07-03-20 09-02-20 10-08-20 12-02-20 12-08-11 12-10-04 12-11-27<br />
07-04-01 09-03-01 10-09-01 12-03-01 12-08-12 12-10-05 12-11-28<br />
07-04-10 09-03-10 10-09-10 12-03-10 12-08-13 12-10-06 12-11-29<br />
07-04-20 09-03-20 10-09-20 12-03-20 12-08-14 12-10-07 12-11-30<br />
07-05-01 09-04-01 10-10-01 12-04-01 12-08-15 12-10-08 12-12-01<br />
07-05-10 09-04-10 10-10-10 12-04-10 12-08-16 12-10-09 12-12-02<br />
07-05-20 09-04-20 10-10-20 12-04-20 12-08-17 12-10-10 12-12-03<br />
07-06-01 09-05-01 10-11-01 12-05-01 12-08-18 12-10-11 12-12-04<br />
07-06-10 09-05-10 10-11-10 12-05-10 12-08-19 12-10-12 12-12-05<br />
07-06-20 09-05-20 10-11-20 12-05-20 12-08-20 12-10-13 12-12-06<br />
07-07-20 09-06-01 10-12-01 12-06-01 12-08-21 12-10-14 12-12-07<br />
07-08-10 09-06-10 10-12-10 12-06-10 12-08-22 12-10-15 12-12-08<br />
07-08-20 09-06-20 10-12-20 12-06-20 12-08-23 12-10-16 12-12-09<br />
07-09-01 09-07-01 11-01-01 12-07-01 12-08-24 12-10-17 12-12-10<br />
07-10-01 09-07-10 11-01-10 12-07-02 12-08-25 12-10-18 12-12-11<br />
07-10-10 09-07-20 11-01-21 12-07-03 12-08-26 12-10-19 12-12-12<br />
07-10-20 09-08-01 11-02-01 12-07-04 12-08-27 12-10-20 12-12-13<br />
07-12-01 09-08-10 11-02-10 12-07-05 12-08-28 12-10-21 12-12-14<br />
07-12-10 09-08-20 11-02-20 12-07-06 12-08-29 12-10-22 12-12-15<br />
08-01-01 09-09-01 11-03-01 12-07-07 12-08-30 12-10-23 12-12-16<br />
08-01-20 09-09-10 11-03-10 12-07-08 12-08-31 12-10-24 12-12-17<br />
08-02-20 09-09-20 11-03-20 12-07-09 12-09-01 12-10-25 12-12-18<br />
08-03-01 09-10-01 11-04-01 12-07-10 12-09-02 12-10-26 12-12-19<br />
08-03-10 09-10-10 11-04-10 12-07-11 12-09-03 12-10-27 12-12-20<br />
08-03-20 09-10-20 11-04-20 12-07-12 12-09-04 12-10-28 12-12-21<br />
08-04-01 09-11-01 11-05-01 12-07-13 12-09-05 12-10-29 12-12-22<br />
08-04-20 09-11-10 11-05-10 12-07-14 12-09-06 12-10-30 12-12-23<br />
08-05-01 09-11-20 11-05-20 12-07-15 12-09-07 12-10-31 12-12-24<br />
08-05-10 09-12-01 11-06-01 12-07-16 12-09-08 12-11-01 12-12-25<br />
08-06-10 09-12-10 11-06-10 12-07-17 12-09-09 12-11-02 12-12-26<br />
08-06-20 09-12-20 11-06-20 12-07-18 12-09-10 12-11-03 12-12-27<br />
08-07-02 10-01-01 11-07-01 12-07-19 12-09-11 12-11-04 12-12-28<br />
08-07-10 10-01-10 11-07-10 12-07-20 12-09-12 12-11-05 2008-10-23<br />
08-07-20 10-01-20 11-07-20 12-07-21 12-09-13 12-11-06 bb.tgz<br />
08-08-01 10-02-01 11-08-01 12-07-22 12-09-14 12-11-07 boot<br />
08-08-10 10-02-10 11-08-10 12-07-23 12-09-15 12-11-08 current<br />
08-08-21 10-02-20 11-08-20 12-07-24 12-09-16 12-11-09 hold<br />
08-09-01 10-03-01 11-09-01 12-07-25 12-09-17 12-11-10<br />
08-09-10 10-03-10 11-09-10 12-07-26 12-09-18 12-11-11<br />
08-09-21 10-03-20 11-09-20 12-07-27 12-09-19 12-11-12<br />
08-10-01 10-04-01 11-10-01 12-07-28 12-09-20 12-11-13<br />
[root@backup1 /data/www/daily]#<br />
</pre><br />
<br />
So we see that everything up to July 2012 has been pruned. To prune July 2012 we do the following:<br />
<br />
<pre>mv 12-07-01 hold<br />
mv 12-07-10 hold<br />
mv 12-07-20 hold<br />
rm -fr 12-07*<br />
mv hold/* .</pre><br />
<br />
== Free up space on Other Servers ==<br />
<br />
Many servers start to run out of disk space over time. Often it is caused by unread mail for<br />
root or log files. <br />
<br />
To find the source of the problems, you use "du" to find where the disk space is being used.<br />
You can't do a du on /proc or /dev, si I use the command<br />
<br />
[root@virt11 /]# du -hs [a-c]* deprecated [e-o]* [q-u]* var | tee duhs0<br />
<br />
which produces something like this.<br />
<br />
4.0K backup<br />
4.0K backup1<br />
4.0K backup2<br />
4.0K backup3<br />
4.0K backup4<br />
7.5M bin<br />
47M boot<br />
4.0K deprecated<br />
92M etc<br />
30M home<br />
8.0K initrd<br />
541M lib<br />
16K lost+found<br />
8.0K media<br />
0 misc<br />
8.0K mnt<br />
0 net<br />
92M opt<br />
336K root<br />
36M sbin<br />
8.0K selinux<br />
8.0K srv<br />
0 sys<br />
4.0K test<br />
16K tmp<br />
1.2G usr<br />
583M var<br />
<br />
In this case it looks like /var is the problem, so<br />
<br />
cd /var<br />
du -hs * | tee duhs9<br />
<br />
Produces<br />
<br />
12K account<br />
2.6M analog-5.32<br />
63M cache<br />
24K db<br />
4.0K duhs<br />
4.0K duhs1<br />
4.0K duhs2<br />
4.0K duhs3<br />
4.0K duhs4<br />
4.0K duhs5<br />
4.0K duhs6<br />
4.0K duhs7<br />
4.0K duhs8<br />
32K empty<br />
8.0K games<br />
16K kerberos<br />
42M lib<br />
8.0K local<br />
36K lock<br />
457M log<br />
0 mail<br />
8.0K nis<br />
8.0K opt<br />
8.0K preserve<br />
8.0K racoon<br />
240K run<br />
18M spool<br />
8.0K tmp<br />
64K vz<br />
0 vzagent<br />
0 vzagent.tmp<br />
16K vzquota<br />
1.2M www<br />
20K yp<br />
<br />
Usually, the problem is in /var/spool or /var/log, due to unread mail<br />
or excessive log files. You can continue to drill down by doing<br />
a "cd <sbdirectory>" and another "du -hs *".</div>75.80.128.147https://wiki.jcihosting.com/index.php?title=VPS_Management&diff=2230VPS Management2020-04-30T20:30:49Z<p>75.80.128.147: /* Ganeti Install a FreeBSD VPS */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
== Moving a server to a different account in the management system ==<br />
<br />
This is a tough one. You gotta touch a bunch of tables in the mysql database.<br />
<pre><br />
systems<br />
subs<br />
ipmap<br />
</pre><br />
<br />
basically, get the sysid and go off that:<br />
<br />
<pre><br />
update systems set cid='newcid' where sysid=xxxxx<br />
update subs set cid='newcid' where sysid=xxxxx<br />
update ipmap set cid='newcid' where sysid=xxxxx<br />
</pre><br />
<br />
You then need to rename the server disk space to change the old ID to the new one.<br />
<br />
For a jail -<br />
<pre><br />
stopjail <hostname><br />
mv 69.55.2XX.XXX-col0<oldid> 69.55.2XX.XXX-col0<newid><br />
mv 69.55.2XX.XXX-col0<oldid>-DIR 69.55.2XX.XXX-col0<newid>-DIR<br />
vi /usr/local/jail/bin/quad1 [Update the ID in the quad file]<br />
startjail <hostname><br />
</pre><br />
<br />
For a virt -<br />
You will have to <br />
* stop the virt <br />
* rename the directories <br />
* modify the config file /vzconf/<veid>.conf <br />
* rename the virt conf file <br />
* restart the new virt.<br />
<br />
that ought to do it..<br />
<br />
<br />
== Re-enable a customer account in the management system ==<br />
<br />
For example to re-instate col03438, In mysql:<br />
<br />
<pre><br />
show tables ;<br />
select * from customers where cid='col03438' ;<br />
update customers set status=1 where cid='col03438' ;<br />
select * from systems where cid='col03438' ;<br />
update systems set stop_date=NULL where cid='col03438' ;<br />
update systems set cancel_date=NULL where cid='col03438' ;<br />
update systems set status=1 where cid='col03438' ;<br />
select * from subs where cid='col03438' ;<br />
update subs set stop_date=NULL where cid='col03438' ;<br />
update subs set status=1 where cid='col03438' <br />
<br />
</pre><br />
<br />
= Ganeti VPS =<br />
<br />
This is how to create a VPS on the Lamphost ganeti cluster.<br />
I have created scripts for creating a FreeBSD VPS and one for<br />
creating a Linux VPS.<br />
<br />
== Ganeti Install a Linux VPS ==<br />
<br />
To create a linux VPS use<br />
<pre><br />
/root/new <name> <disk size> <memory> <IP Addr> [<OS>]<br />
<br />
( for example /root/new vps.example.com 30 2 69.55.238.10 centos-8 )<br />
</pre><br />
<br />
You need to customize the install by adding a "jci" account and a customer<br />
account (col0XXXX) as well. Make sure the customer acct (col0XXXX) is<br />
in a group to allow sudo.<br />
<br />
== Ganeti Install a FreeBSD VPS ==<br />
<br />
To create a FreeBSD VPS, we have to do a full install. This will <br />
create a new VPS.<br />
<br />
<pre><br />
/root/newfreebsd12 <name> <disk size> <memory> <IP Addr> [<OS>]<br />
</pre><br />
<br />
You will then have to do the install via VNC.<br />
<br />
<br><br />
You need to customize the install by adding a "jci" account and a customer <br />
account (col0XXXX) as well. Make sure the customer acct (col0XXXX) is in a <br />
group to allow sudo. <br />
<br />
<br> <br />
You will need to install sudo.<br />
<br />
<pre> <br />
pkg install sudo # install sudo package<br />
visudo # to add wheel account to sudoers<br />
vi /etc/group # add jci and user acct to wheel<br />
</pre><br />
<br />
<br><br />
You will also have to set up a console in /etc/ttys.<br />
<pre><br />
vi /etc/ttys<br />
</pre><br />
<br />
== Ganeti restart a VPS ==<br />
<br />
To shutdown a ganeti VPS<br />
<br />
<pre><br />
gnt-instance reboot <VPSname><br />
</pre><br />
<br />
<br />
== Ganeti shutdown a VPS ==<br />
<br />
To shutdown a ganeti VPS<br />
<br />
<pre><br />
gnt-instance shutdown <VPSname><br />
</pre><br />
<br />
To permanently delete a ganeti VPS<br />
<br />
<pre><br />
gnt-instance remove <VPSname><br />
</pre><br />
<br />
== Ganeti Re-Intsall a VPS ==<br />
<br />
Re-installing the OS will destroy all data on the VPS.<br />
Login to the Ganeti master node (currently gn1).<br />
<br />
To reinstall the OS on a VPS the VPS must be shutdown.<br />
<pre><br />
gnt-instance shutdown <VPSname><br />
</pre><br />
<br />
Do the reinstall.<br />
<pre><br />
gnt-instance reinstall <VPSname><br />
</pre><br />
<br />
Restart the VPS.<br />
<pre><br />
gnt-instance startup <VPSname><br />
</pre><br />
<br />
Now you need to do the customization for the VPS. This is<br />
basically the same as the Dedicated server customization.<br />
<br />
https://69.55.229.8/index.php/New_Signups#Dedicated_Server_Install_Checklist<br />
<br />
= FreeBSD 10.1 VPS (jail5 and jail6) =<br />
<br />
This is information on how to manage the FreeBSD 10.1 VPS servers on Jail5.<br />
<br />
<br />
== Making new customer VPS ==<br />
<pre><br />
virtmake IP[,IP] <CID> hostname ipfw# email {FSP|FB|FS|FM|FP} [size in GB]<br />
</pre><br />
<br />
Note: <br />
For now, CID must be unique in the last 2 digits of the CID.<br />
<br />
Must do a "vm start <CID>" after creating a VPS.<br />
<br />
== Starting VPS ==<br />
To start all vms on the system<br />
<pre><br />
vm start<br />
</pre><br />
<br />
To start a single VPS <br />
<pre><br />
vm start <CID><br />
</pre><br />
<br />
== List running VPS ==<br />
<pre><br />
vm status<br />
</pre><br />
<br />
== Attach to a user VPS console ==<br />
<pre><br />
vm attach <CID><br />
</pre><br />
<br />
<br />
To detach from a console, type tilde control-d (~ ctl-d).<br />
<br />
== Stop a VPS ==<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
== Restart a VPS ==<br />
<pre><br />
vm restart <CID><br />
</pre><br />
<br />
<br />
== Modify a VPS ==<br />
<br />
To modify a VPS config, stop the VPS<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
Mount the VPS data file<br />
<pre><br />
vm mount <CID><br />
<br />
cd /data/vm/private/<CID>/mount<br />
</pre><br />
<br />
Modify the user files necessary.<br />
<br />
Restart the VPS.<br />
<br />
<pre><br />
vm umount <CID><br />
<br />
vm start <CID><br />
</pre><br />
<br />
== Notes on a new FBSD 10.x VPS (jail6) ==<br />
<br />
<pre><br />
#! /bin/sh<br />
#<br />
# new <colo#> <disk><br />
<br />
<br />
memory=1024<br />
<br />
x="$1"<br />
<br />
disk="$2"<br />
if [ "y" == "y$disk" ] ; then<br />
disk=60<br />
fi<br />
<br />
disk2=`expr $disk - 4`<br />
<br />
<br />
vm image provision 04dcdf96-d843-11e6-b131-001e67fcc64f col0$x<br />
# vm configure col0$x<br />
sed -e "s/memory=256M/memory=${memory}M/" /vms/col01114/col01114.conf >/vms/col0$x/col0$x.conf<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
# vm start col0$x<br />
<br />
<br />
exit<br />
<br />
vm console col0$x<br />
<br />
#<br />
# to increase the customer disk<br />
#<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
<br />
## on the customer VPS<br />
<br />
gpart recover vtbd0<br />
gpart show vtbd0<br />
swapoff /dev/vtbd0p3<br />
gpart delete -i 3 vtbd0<br />
gpart show vtbd0<br />
sysctl kern.geom.debugflags=16<br />
gpart resize -i 2 -a 4k -s $disk2\G vtbd0<br />
<br />
growfs /dev/vtbd0p2<br />
<br />
gpart show vtbd0<br />
gpart add -t freebsd-swap -a 4k vtbd0<br />
gpart show vtbd0<br />
swapon /dev/vtbd0p3<br />
<br />
df -h<br />
reboot<br />
<br />
vi /etc/rc.conf<br />
hostname<br />
IP Addr<br />
passwd root<br />
passwd user<br />
clear history<br />
cp /dev/null /var/log/utx.log<br />
cp /dev/null ~root/.history<br />
history -c<br />
<br />
</pre><br />
<br />
= FreeBSD Jails =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Making new customer jail 4.x ==<br />
<br />
1. run [[#js|js]] to figure out which partition, IP to put it on, find an unused vn, and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vnX hostname shorthost quad/safe# ipfw# email [gb disk, default 4]</tt><br />
<br />
jail14# /tmp/jailmake 69.55.226.152 /mnt/data2/69.55.226.152-col00182 vn23 negev.cerastes.org negev 4 182 cerastes@cerastes.org<br />
<br />
<br />
== Making new customer jail 6.x ==<br />
<br />
1. run [[#js|js]] to figure out which gvinum vols are free (and which mnt (disk) each gvinum goes with/to), IP to put it on and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vN[,vN] hostname shorthost quad/safe# ipfw# email</tt><br />
<br />
jail19# jailmake 69.55.236.17 /mnt/data1/69.55.236.17-col01574 v10 uncle-leo.commadev.com uncle-leo 1 1574 lee@commadev.com,lee@gmail.com<br />
<br />
<br />
== Changing an ip for freebsd VPS ==<br />
<br />
*stopjail (hostname)<br />
*on systems using vnfile or mdfile: rename vnfile/mdfile, using new ip<br />
*rename dir using new ip<br />
*Edit quad (make sure to update all lines: directory, vn/mdfile)<br />
*buildsafe (not necessary on systems having only quad1)<br />
*if you're adding an ip not already on the host machine: <tt>ipadd x.x.x.x</tt><br />
*startjail (hostname)<br />
if backups: <tt>mvbackups</tt> ...<br />
*Edit dir/ip in mgmt<br />
*if any rules, update any firewall rules to use new ip (careful not to make sure there arent multiple rules per ipfw#- search by colo#)<br />
*(if customer asks and has domains) update their domains on ns1c<br />
*optional: update ipfw counters on the host server<br />
<br />
<br />
== Rename a gconcat vol ==<br />
<br />
*stop jail, unmount everything<br />
*gconcat stop vol<br />
*gconcat clear /dev/gvinum/vN<br />
*gconcat clear /dev/gvinum/vN+1<br />
*gconcat label -v newname /dev/gvinum/vN /dev/gvinum/vN+1<br />
*bsdlabel /dev/concat/newname<br />
(make sure a partition is not fstype unused, if so change to: 4.2BSD)<br />
<br />
<br />
== Remaking a VPS (on same jail) ==<br />
<br />
# take him down (stopjail)<br />
# make note of host, vn/gvin/md, ip<br />
# remove from quad/safe<br />
# move vnfile (rename)<br />
# remove dir (jailmake will make a new one)<br />
# remove ipfw counts<br />
# jailmake<br />
# update backup script<br />
# update db with new dir/ip (use the “move” link, mark old as stopped today)<br />
# update firewall if changed ip<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
<br />
== Adding disk to a 4.x jail ==<br />
<br />
NOTE: if you unmount before vnconfig, re-vnconfig then unmount then vnunconfig<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them.<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to dump to a different partition and create the new vn on the current partition. If no space to make make dump, then move old vn’s to backup or worse, dump to backup.<br />
<br />
<br />
2. <tt>g <customerID></tt> to get the info<br />
<br />
<br />
3. make vnfile, newfs and mount. USE CAUTION not to overwrite existing vn. <br><br />
Ex: <tt>vnconfig –T -S 9g -s labels -c /dev/vn22 /mnt/data2/65.214.160.117-col00496<br><br />
disklabel -r -w vn30 auto<br><br />
newfs /dev/vn30c<br><br />
mkdir /mnt/data2/65.214.160.117-col00496-DIR<br><br />
mount /dev/vn30c /mnt/data2/69.55.237.26-col00241-DIR<br><br />
cd /mnt/data2/65.214.160.117-col00496-DIR</tt><br />
<br />
<br />
4. <tt>jailkill <hostname></tt><br />
<br />
<br />
5. <tt>dump -0a -f - /dev/vn1 | restore -r -f -</tt> <br />
<br />
<br />
6. <tt>rm restoresymtable</tt><br />
<br />
<br />
7. unmount and unvnconfig old system:<br><br />
<tt>umount /dev/vnNNc <br><br />
vnconfig –u /dev/vnNN</tt><br />
<br />
<br />
8. move vnfile. Ex: <tt>mv /mnt/data1/69.55.237.26-col00241 /mnt/data1/old-col00241-vnfile-20110312-noarchive</tt><br />
<br />
<br />
9. remove old dir: <tt>rmdir /mnt/data1/69.55.237.26-col00241-DIR</tt><br />
<br />
<br />
10. edit quad/safe to point to new location, run <tt>[[#buildsafe|buildsafe]]</tt><br />
<br />
<br />
11. start jail: <tt>[[#startjail|startjail]] <hostname></tt><br />
<br />
<br />
12. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
13. mv backups if necessary<br />
<br />
WARNING: if you restore to unmounted vn then you’re actually restoring to /mnt/data1<br />
you can tar and do above to recover<br />
<br />
NOTE: you can’t move a customer to a system that has a base sys that is diff from the one they came from (BSD 4.8 to 4.5 = won’t work)<br />
<br />
NOTE: we don’t mount procfs's to anyone’s system by default.<br />
<br />
NOTE: mount_nulls: <br />
also goofy mount_nulls seen in df -k on jail2<br><br />
also needs to be done in safe and quad<br><br />
if he wants another, use safe/quad as example and add it to file. also exec command at prompt to add mount (no reboot, remount or rejail necessary)<br />
<br />
== Moving customer to another jail machine ==<br />
<br />
Systems should only be moved between similarly-versioned jails. If you cannot get a match, you should go to freebsd.org and see what the differences were between the version you're moving from and the new version. Many of the updates will be to drivers and kernels - these don't affect the customer. Perhaps they updated traceroute or a library...you should copy the new/changed files into the VPS/jail on the new host. An alternate method for moving to a different version would be to buildworld to bring their full distribution up to date. Optional steps for this kind of upgrade are included below.<br />
<br />
1. <tt>g <customerID></tt><br />
<br />
<br />
2. <tt>[[#jailkill|jailkill]] <hostname></tt><br />
<br />
<br />
3. create new device on target system<br><br />
4.x:<br><br />
<pre>vnconfig –T –S 4g –s labels –c /dev/vn1 /mnt/data1/69.55.22x.x-col00XXX<br />
disklabel –r –w vn1 auto<br />
newfs /dev/vn1c</pre><br />
<br />
6.x:<br><br />
<pre>bsdlabel -r -w /dev/gvinum/v1<br />
newfs /dev/gvinum/v1a<br />
or <br />
gconcat label v1-v3 /dev/gvinum/v1 /dev/gvinum/v2 /dev/gvinum/v3<br />
bsdlabel -r -w /dev/concat/v1-v3<br />
newfs /dev/concat/v1-v3a</pre><br />
<br />
7.x+:<br><br />
Run jailmakeempty then skip step 4 & 7 below<br />
<br />
<br />
4. make and mount the dir on new system. <br><br />
Ex: <pre>mkdir /mnt/data2/69.55.230.3-col00123-DIR<br />
chmod 755 /mnt/data2/69.55.230.3-col00123-DIR<br />
mount [device] /mnt/data2/69.55.230.3-col00123-DIR</pre><br />
<br />
<br />
5. [[#stopjail|stopjail]] <hostname> 1<br />
<br />
<br />
6. dump fs to new system. ex: <tt>[[#dumpremoterestore|dumpremoterestore]] /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR</tt> (make sure you can ssh as root on remote machine) <br />
<br />
<br />
6a. OPTIONAL BUILDWORLD:<br><br />
cd /usr/src<br />
make world DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
cd etc<br />
make distribution DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
(you may have to rm an openssh file and re-make dist)<br />
<br />
rm -rf /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/periodic/daily/400.status-disks<br />
vi /etc/periodic/security/100.chksetuid<br />
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt><br><br />
with: <tt>MP='/' (use single quotes)</tt><br />
<br />
Ask user if they want ports overwritten with current, if yes:<br />
cp -r /usr/ports /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/usr<br />
<br />
on source: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d .<br />
(copy in all info)<br />
<br />
on target: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d . <br />
(paste all info)<br />
<br />
on source: <br />
cat /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group<br />
(copy in all info)<br />
<br />
on target:<br />
cat > /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group <br />
(paste all info)<br />
<br />
<br />
7. edit quad on source system copy over entries for the jail to the target system <br />
take care that the vn/gvinum/gconcat/md devices on the target system are’t in use and that the /mnt/dataN path doesn’t need changing<br />
<br />
<br />
8. run [[#buildsafe|buildsafe]] (if this is <=6.x) on target system (copies changes made to quad into safe file)<br />
<br />
<br />
9. remove ip from source system. Ex: <tt>ipdel 69.55.230.3</tt><br />
<br />
<br />
10. add ip to target system. Ex: <tt>ipadd 69.55.230.3</tt><br />
<br />
<br />
11. start new system, with [[#startjail|startjail]] or manually, pasting entries found from running <tt>g <customerID></tt> on the new system<br />
<br />
<br />
12. run <tt>[[#canceljail|canceljail]] col0xxxx</tt> on source system. This should prompt you to remove backups if any existed. Do not let it do this, rather:<br />
<br />
<br />
13. if backups existed move them to the new host via the <tt>[[#mvbackups_.28freebsd.29|mvbackups]] script</tt><br />
<br />
<br />
14. edit quad on source system, edit comment to reflect a move rather than a cancel, ex: <tt># moved to jail2 col00241</tt><br />
<br />
<br />
15. edit mgmt to reflect new host and dir for new system<br />
<br />
<br />
16. optional add ipfw rules on new system. Ex:<br />
<tt>ipfw add 01231 count ip from 69.55.230.2 to any<br />
ipfw add 01232 count ip from any to 69.55.230.2</tt><br />
<br />
== Increasing inodes for a VPS ==<br />
<br />
when doing the newfs: <br />
newfs -i 4096 ...<br />
<br />
<br />
== Jail services unable to contact services within same jail ==<br />
<br />
This is due to a messed up routing table (only seen it on jail2 before). <br />
<br />
route delete 69.55.228.65/32 route add -net 69.55.228.65/32 -iface fxp0 -nostatic -cloning route add 69.55.228.65 -iface lo0<br />
<br />
<br />
== /dev/null permission resets == <br />
<br />
Applies to 4.x jails only<br />
<br />
For some reason that I do not understand at all, the /dev/null node in customer jails, after the system is restarted, often reverts permissions to 0600 ... which is bad, because non-root processes cannot redirect to /dev/null anymore. In fact, a fair number of server daemons will complain and/or fail if this is the case.<br />
<br />
So, after a restart, after all systems have restarted, run postboot, which does something like:<br />
<br />
for f in `df -k | grep /dev/vn | awk '{print $6}'` ; do chmod 0666 $f/dev/null ; done<br />
<br />
Later, after the system is up, if anyone ever complains about their /dev/null permissions for any reason ... really if anyone mentions<br />
/dev/null in any way, tell them to run:<br />
<br />
chmod 0666 /dev/null<br />
<br />
We have ultimately fixed this by adding chmods to the quad/safe's<br />
<br />
<br />
== mknod in a jail == <br />
<br />
For a customer who wants to run a chroot name server (named) in a jail.<br />
<br />
<pre><br />
cd /mnt/data1/<user directory>/var/named<br />
mknod -c 1 1 null<br />
mknod -c 2 3 random<br />
</pre><br />
<br />
== Postfix Problems ==<br />
<br />
This hasn't happened in a long time, probably no longer valid with modern OS's<br />
<br />
Postfix is an alternate MTA - people replace sendmail with it in much the same way that many people also replace sendmail with qmail. The problem is that if you install postfix inside of a jail, by default it will not work properly - the user will see error messages like this:<br />
<br />
incoming mail generates errors like this:<br />
<br />
mail_queue_enter: create file incoming/298266.55902: File too large<br />
<br />
and further, sending mail generates errors like this:<br />
<br />
postdrop: warning: mail_queue_enter: create file maildrop/627930.56676: File too large<br />
<br />
This is very easy to solve. In fact, the FreeBSD welcome email that is sent out by `jailmake` now contains this block of text:<br />
<br />
- if you plan on installing postfix, email us first and ask for a necessary patch.<br />
<br />
So, anytime you see a support email that says anything about postfix not working, paste that line from the welcome email into your response to them, and right away, forward the postfix patch to them in an email.<br />
<br />
Here are the instructions:<br />
<br />
Simply copy the patch into /usr/ports/mail/postfix/files, rename the file to ‘patch-file_limit.c’ and then cd to /usr/ports/mail/postfix and run `make install`.<br />
<br />
The postfix patch is the very first email in the support email box, and we never delete or save it because we always want it there to forward to people. The instructions in the email are very clear, and it solves their problem 100% of the time.<br />
<br />
<br />
== Problems with `find` ==<br />
<br />
There was a bug in the FreeBSD vn-filesystem code. The bug causes the system to crash or hang when someone runs the `find` command inside their vn-backed filesystem (and even though the bug may have been fixed, due to the i/o overhead, we still don't like find).<br />
<br />
Now, this does not always happen - however, any jail machine with more than 20 systems on it will crash every night if every system on it runs the daily periodic script out of their crontab, and updates their locate database with the find command.<br />
<br />
(the locate database is a small, FreeBSD specific database that is populated nightly from a cron job, and is then used to provide fast answers to the `locate` command)<br />
<br />
In normal systems (32-40 jails on them), all that has to be done is make sure nobody runs the daily periodic. This means that the file /etc/crontab, instead of looking like this:<br />
<br />
<pre># do daily/weekly/monthly maintenance<br />
1 3 * * * root periodic daily<br />
15 4 * * 6 root periodic weekly<br />
30 5 1 * * root periodic monthly</pre><br />
<br />
Needs to look like this:<br />
<br />
<pre># DO NOT UNCOMMENT THESE - contact support@johncompanies.com for details<br />
#1 3 * * * root periodic daily<br />
#15 4 * * 6 root periodic weekly<br />
#30 5 1 * * root periodic monthly<br />
## DO NOT UNCOMMENT THESE ^^^^^^^^</pre><br />
<br />
and the problem will generally not occur at all. New filesystem images are always altered in this way, so all systems are set like this.<br />
<br />
However, if you have enough systems running, even casual uses of find, outside of the periodic scripts, can cause the system to crash. We are seeing this on jail13, which crashes about once per week.<br />
<br />
There is not yet any resolution to this problem.<br />
<br />
So, if you have a freebsd system crash, see if anyone has uncommented their daily periodic line - you can see them for everyone by running:<br />
<br />
/bin/sh<br />
<br />
and then running:<br />
<br />
<pre>for f in `df -k | grep vn | awk '{print $6}'` ; do echo `cat $f/etc/crontab | grep "periodic daily"` $f ; done | more</pre><br />
<br />
(all one line, one command)<br />
<br />
And then edit those /etc/crontabs and comment them out again.<br />
<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
* if you are getting disk full messages for a BSD customer, it's fairly safe to clear out their /usr/ports/distfiles dir<br />
* 4.x: ps and top an only be run by root in these jails. Done on purpose:<br />
As for `ps` and `top` - non-root users can run them, just not successfully because we have locked the permissions on /dev/mem and /dev/kmem to be root-readable only. That is why non-root users cannot successfully run `ps` and `top`.<br />
* user quotas do not work on freebsd jails - you cannot set up quotas at all, and that's that.<br />
* You cannot inject a process into a 4.x jail, only HUP running processes. 6.x and onward you can with jexec<br />
* jails see base machine’s uptime/load when running top/w<br />
* if someone is unable to get in- cant ping, etc- see if they were blocked by castle (DoS), see if their ip is on the system (post reboot it was lost cause wasn’t in the rc.conf). preboot should catch that.<br />
* in FreeBSD you cant su to root unless you belong to wheel group – so if you remove your acct and setup a new one, we have to add it to wheel group (add to /etc/group)<br />
* Dmesg from underlying sys is seen in customers dmesg on jail<br />
* Popper process going crazy @ 40% for 10min = someone who leaves mail on server<br />
* Don’t force umounts on 4.x jails – it crashes the machine, generally ok on newer machines (running md)<br />
* Good book for admin http://search.barnesandnoble.com/booksearch/isbnInquiry.asp?userid=t824VyRAYz&isbn=0596005164&itm=2<br />
* Self-sign ssl cert http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#selfcert<br />
* conversation with Glenn about semaphores and pgsql<br />
<pre>SDBoody: hey, these are valid amounts/figures/increments right:<br />
kern.ipc.semmni=1280<br />
kern.ipc.semmns=1280<br />
<br />
gr8feen: probably... I always forget exactly what those are, so I usually have to look them up<br />
SDBoody: semaphores<br />
SDBoody: i took the current 1024 and added 256 to them<br />
SDBoody: need more for pgsql<br />
gr8feen: I meant the mni and mns parts...<br />
gr8feen: some of those are not ones you just want to add to... hang on a sec and I'll look them up..<br />
gr8feen: what's semmsl set to?<br />
SDBoody: kern.ipc.semmsl: 1024<br />
<br />
SDBoody: kern.ipc.msgseg: 2048 <br />
kern.ipc.msgssz: 8 <br />
kern.ipc.msgtql: 40 <br />
kern.ipc.msgmnb: 2048 <br />
kern.ipc.msgmni: 40 <br />
kern.ipc.msgmax: 16384 <br />
kern.ipc.semaem: 16384 <br />
kern.ipc.semvmx: 65534 <br />
kern.ipc.semusz: 152 <br />
kern.ipc.semume: 10 <br />
kern.ipc.semopm: 100 <br />
kern.ipc.semmsl: 1024 <br />
kern.ipc.semmnu: 512 <br />
kern.ipc.semmns: 1024 <br />
kern.ipc.semmni: 1024 <br />
kern.ipc.semmap: 768 <br />
kern.ipc.shm_allow_removed: 0 <br />
kern.ipc.shm_use_phys: 1 <br />
kern.ipc.shmall: 262144 <br />
kern.ipc.shmseg: 256 <br />
kern.ipc.shmmni: 784 <br />
kern.ipc.shmmin: 1 <br />
kern.ipc.shmmax: 536870912 <br />
kern.ipc.maxsockets: 25600 <br />
<br />
gr8feen: ok...msl is max per id, mni is max ids, mns is max number of semaphores... so you probably want something like mns = mni * msl<br />
gr8feen: which one did you run out of?<br />
SDBoody: not sure how to tell- ipcs shows the sems in use add up to 1024<br />
SDBoody: there are 59 entries<br />
gr8feen: I'm assuming you tried to start postgres and it failed?<br />
SDBoody: yes<br />
gr8feen: it should have logged why, somewhere..<br />
gr8feen: if I recall, it'll tell you which one it ran out of<br />
SDBoody: > DETAIL: Failed system call was semget(1, 17, 03600).<br />
<br />
gr8feen: so it wanted an id with 17 semaphores...I'd start by making mns = 17*mni and leave mni and mnl set to what they are now and see what it does<br />
SDBoody: i think mni is plenty high<br />
SDBoody: ok, more reasonable 17408<br />
gr8feen: yeah...just change that one and see how it goes..<br />
SDBoody: and leave mni alone at 1024?<br />
gr8feen: yeah...mni id the max number of ids...but if it's trying to get something like 17 per id, your going to hit mns before you hit anything else<br />
SDBoody: right, but doesn't hurt to have it that high (assuming)<br />
<br />
gr8feen: not really... I think those get allocated out of ram that you cant page out, but it's still such a small amount that it really doesn't matter<br />
SDBoody: looks like that worked, thx!<br />
gr8feen: cool<br />
<br />
see semaphores: ipcs -a -s</pre><br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
== Making new customer VE (vm) ==<br />
<br />
This applies only to new virts >= 4.x<br />
<br />
grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.<br />
<br />
1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
also, ping the IP address.<br />
<br />
2. put CT on whichever partition has more space<br />
df -h<br />
<br />
3. vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> ) <br />
vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@johncompanies.com centos-6-x86_64 LM<br />
<br />
4. copy veid, dir, ip and password to pending customer screen. activate customer<br />
<br />
== Making new customer VE (vemakexxx) ==<br />
<br />
This applies to older virts with old templates. This should probably not be used at all anymore.<br />
<br />
<br />
1. look thru hist for ip<br />
<br />
2. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
<br />
3. put ve on whichever partition has more space<br />
vemakerh9 <veid> <ip> <hostname> <mount> <email> [gb disk]; <256|384|512> <veid><br />
vemakerh9 866 69.55.226.109 ngentu.com /vz1 ayo@ngantu.com,asd@asd.com 5; 256 866<br />
<br />
4. copy (veid), dir, and ip to pending customer screen (pass set to p455agfa)<br />
<br />
<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrateonline/migrate failures (migrate manually) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
You may also see migration failing due to quota issues.<br />
<br />
You can try to resolve by copying any quota file into the file you need:<br />
<br />
cp /var/vzquota/quota.1 /var/vzquota/quota.xxx<br />
<br />
If it complains about quota running you should then be able to stop it<br />
<br />
vzquota off xxxx<br />
<br />
If all else fails, migrate to a new VEID<br />
i.e. 1234 becomes 12341<br />
<br />
<br />
If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
'''Note:''' don't forget to fix /usr/local/sbin/backup.config for new disk location<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
=== Cant change file /etc/sysconfig/network ===<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
=== What template is user using ===<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
=== Edits needed to run java ===<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
=== Set time zone ===<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
=== Failing shm_open calls ===<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
=== Debian can't get on the network ===<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
=== CentOS can't get on the network ===<br />
<br />
When CentOS system was running, but can't get on the network, they probably updated iproute via yum.<br />
<br />
Copy an older version of the iproute rpm to their VPS.<br />
<br />
<pre>cp -p /root/iproute/iproute-2.6.32-23.el6.<i686>.rpm /vz2/private/<1527>/fs/root</pre><br />
<br />
Enter their ve, install rpm, restart network, and prevent yum from updating iproute.<br />
<br />
<pre><br />
ve <1527><br />
rpm -i --force iproute-2.6.32-23.el6.<i686>.rpm<br />
rpm -q iproute<br />
rpm -e <new iproute rpm><br />
service network restart<br />
echo "exclude=iproute" >>/etc/yum/ip-exclude<br />
echo "include=/etc/yum/ip-exclude" >>/etc/yum.conf<br />
</pre><br />
<br />
<br />
<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
----<br />
<br />
* uptime/load that vz users see isn’t what we see- it's their own load<br />
* Vz overwrites resolve.conf on every boot to what’s in ve config file. Also overwrites hostname<br />
* Quotas in vz: set Gid_limit in vz config file and restart ve. Errors leading up to this will be out of files but no inode or space issues. set to 2000 in conf, and restart vz<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
To create a new Linux VPS<br />
<br />
* get an IP address from mgmt screen<br />
* run df -h to figure out which disk has enough space<br />
* run vm with no arguments to get syntax and exact template name<br />
* use col0xxxx for hostname if they don’t give you a hostname<br />
* copy over dir, veid, ip and password to pending customer screen<br />
<br />
vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> )<br />
<br />
== disable ==<br />
To disable a customers Virtuozzo VPS server.<br />
<br />
* stop the server<br />
vzctl stop <veid><br />
* prevent restart on server reboot and user restart via Control Panel<br />
vzctl set <veid> --disabled=yes --offline_management=no --save<br />
<br />
== cancelve ==<br />
When a customer cancels all service.<br />
<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
<br />
display the contents of the vz config file for the <veid><br />
<br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
<br />
List all virtual environments running on this machine.<br />
<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
<br />
show processes running under this <veid>.<br />
<br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
<br />
Restart a <veid>.<br />
<br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
<br />
enter a <veid> as root.<br />
<br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
<br />
execute a single command in the <veid> environment.<br />
<br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem VEID <256|512|768|1024|1280|1536|2048> [burst GB]<br />
adjusts the memory resources for the VE. If this is on VZ ver >4, you will also be prompted to provide a burst ram amount. Sample usage:<br />
<br />
<pre># setmem 21671 1024 4<br />
/usr/sbin/vzctl set 21671 --ram 1024m --swap 3072m --save<br />
</pre><br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>75.80.128.147https://wiki.jcihosting.com/index.php?title=VPS_Management&diff=2229VPS Management2020-04-30T20:28:58Z<p>75.80.128.147: /* Ganeti Install a Linux VPS */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
== Moving a server to a different account in the management system ==<br />
<br />
This is a tough one. You gotta touch a bunch of tables in the mysql database.<br />
<pre><br />
systems<br />
subs<br />
ipmap<br />
</pre><br />
<br />
basically, get the sysid and go off that:<br />
<br />
<pre><br />
update systems set cid='newcid' where sysid=xxxxx<br />
update subs set cid='newcid' where sysid=xxxxx<br />
update ipmap set cid='newcid' where sysid=xxxxx<br />
</pre><br />
<br />
You then need to rename the server disk space to change the old ID to the new one.<br />
<br />
For a jail -<br />
<pre><br />
stopjail <hostname><br />
mv 69.55.2XX.XXX-col0<oldid> 69.55.2XX.XXX-col0<newid><br />
mv 69.55.2XX.XXX-col0<oldid>-DIR 69.55.2XX.XXX-col0<newid>-DIR<br />
vi /usr/local/jail/bin/quad1 [Update the ID in the quad file]<br />
startjail <hostname><br />
</pre><br />
<br />
For a virt -<br />
You will have to <br />
* stop the virt <br />
* rename the directories <br />
* modify the config file /vzconf/<veid>.conf <br />
* rename the virt conf file <br />
* restart the new virt.<br />
<br />
that ought to do it..<br />
<br />
<br />
== Re-enable a customer account in the management system ==<br />
<br />
For example to re-instate col03438, In mysql:<br />
<br />
<pre><br />
show tables ;<br />
select * from customers where cid='col03438' ;<br />
update customers set status=1 where cid='col03438' ;<br />
select * from systems where cid='col03438' ;<br />
update systems set stop_date=NULL where cid='col03438' ;<br />
update systems set cancel_date=NULL where cid='col03438' ;<br />
update systems set status=1 where cid='col03438' ;<br />
select * from subs where cid='col03438' ;<br />
update subs set stop_date=NULL where cid='col03438' ;<br />
update subs set status=1 where cid='col03438' <br />
<br />
</pre><br />
<br />
= Ganeti VPS =<br />
<br />
This is how to create a VPS on the Lamphost ganeti cluster.<br />
I have created scripts for creating a FreeBSD VPS and one for<br />
creating a Linux VPS.<br />
<br />
== Ganeti Install a Linux VPS ==<br />
<br />
To create a linux VPS use<br />
<pre><br />
/root/new <name> <disk size> <memory> <IP Addr> [<OS>]<br />
<br />
( for example /root/new vps.example.com 30 2 69.55.238.10 centos-8 )<br />
</pre><br />
<br />
You need to customize the install by adding a "jci" account and a customer<br />
account (col0XXXX) as well. Make sure the customer acct (col0XXXX) is<br />
in a group to allow sudo.<br />
<br />
== Ganeti Install a FreeBSD VPS ==<br />
<br />
To create a FreeBSD VPS, we have to do a full install. This will <br />
create a new VPS.<br />
<br />
<pre><br />
/root/newfreebsd12 <name> <disk size> <memory> <IP Addr> [<OS>]<br />
</pre><br />
<br />
You will then have to do the install via VNC.<br />
<br />
<br><br />
You need to customize the install by adding a "jci" account and a customer<br />
account (col0XXXX) as well. <br />
<br />
<br> <br />
You will need to install sudo.<br />
<br />
<pre> <br />
pkg install sudo # install sudo package<br />
visudo # to add wheel account to sudoers<br />
vi /etc/group # add jci and user acct to wheel<br />
</pre><br />
<br />
<br><br />
You will also have to set up a console in /etc/ttys.<br />
<pre><br />
vi /etc/ttys<br />
</pre><br />
<br />
== Ganeti restart a VPS ==<br />
<br />
To shutdown a ganeti VPS<br />
<br />
<pre><br />
gnt-instance reboot <VPSname><br />
</pre><br />
<br />
<br />
== Ganeti shutdown a VPS ==<br />
<br />
To shutdown a ganeti VPS<br />
<br />
<pre><br />
gnt-instance shutdown <VPSname><br />
</pre><br />
<br />
To permanently delete a ganeti VPS<br />
<br />
<pre><br />
gnt-instance remove <VPSname><br />
</pre><br />
<br />
== Ganeti Re-Intsall a VPS ==<br />
<br />
Re-installing the OS will destroy all data on the VPS.<br />
Login to the Ganeti master node (currently gn1).<br />
<br />
To reinstall the OS on a VPS the VPS must be shutdown.<br />
<pre><br />
gnt-instance shutdown <VPSname><br />
</pre><br />
<br />
Do the reinstall.<br />
<pre><br />
gnt-instance reinstall <VPSname><br />
</pre><br />
<br />
Restart the VPS.<br />
<pre><br />
gnt-instance startup <VPSname><br />
</pre><br />
<br />
Now you need to do the customization for the VPS. This is<br />
basically the same as the Dedicated server customization.<br />
<br />
https://69.55.229.8/index.php/New_Signups#Dedicated_Server_Install_Checklist<br />
<br />
= FreeBSD 10.1 VPS (jail5 and jail6) =<br />
<br />
This is information on how to manage the FreeBSD 10.1 VPS servers on Jail5.<br />
<br />
<br />
== Making new customer VPS ==<br />
<pre><br />
virtmake IP[,IP] <CID> hostname ipfw# email {FSP|FB|FS|FM|FP} [size in GB]<br />
</pre><br />
<br />
Note: <br />
For now, CID must be unique in the last 2 digits of the CID.<br />
<br />
Must do a "vm start <CID>" after creating a VPS.<br />
<br />
== Starting VPS ==<br />
To start all vms on the system<br />
<pre><br />
vm start<br />
</pre><br />
<br />
To start a single VPS <br />
<pre><br />
vm start <CID><br />
</pre><br />
<br />
== List running VPS ==<br />
<pre><br />
vm status<br />
</pre><br />
<br />
== Attach to a user VPS console ==<br />
<pre><br />
vm attach <CID><br />
</pre><br />
<br />
<br />
To detach from a console, type tilde control-d (~ ctl-d).<br />
<br />
== Stop a VPS ==<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
== Restart a VPS ==<br />
<pre><br />
vm restart <CID><br />
</pre><br />
<br />
<br />
== Modify a VPS ==<br />
<br />
To modify a VPS config, stop the VPS<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
Mount the VPS data file<br />
<pre><br />
vm mount <CID><br />
<br />
cd /data/vm/private/<CID>/mount<br />
</pre><br />
<br />
Modify the user files necessary.<br />
<br />
Restart the VPS.<br />
<br />
<pre><br />
vm umount <CID><br />
<br />
vm start <CID><br />
</pre><br />
<br />
== Notes on a new FBSD 10.x VPS (jail6) ==<br />
<br />
<pre><br />
#! /bin/sh<br />
#<br />
# new <colo#> <disk><br />
<br />
<br />
memory=1024<br />
<br />
x="$1"<br />
<br />
disk="$2"<br />
if [ "y" == "y$disk" ] ; then<br />
disk=60<br />
fi<br />
<br />
disk2=`expr $disk - 4`<br />
<br />
<br />
vm image provision 04dcdf96-d843-11e6-b131-001e67fcc64f col0$x<br />
# vm configure col0$x<br />
sed -e "s/memory=256M/memory=${memory}M/" /vms/col01114/col01114.conf >/vms/col0$x/col0$x.conf<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
# vm start col0$x<br />
<br />
<br />
exit<br />
<br />
vm console col0$x<br />
<br />
#<br />
# to increase the customer disk<br />
#<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
<br />
## on the customer VPS<br />
<br />
gpart recover vtbd0<br />
gpart show vtbd0<br />
swapoff /dev/vtbd0p3<br />
gpart delete -i 3 vtbd0<br />
gpart show vtbd0<br />
sysctl kern.geom.debugflags=16<br />
gpart resize -i 2 -a 4k -s $disk2\G vtbd0<br />
<br />
growfs /dev/vtbd0p2<br />
<br />
gpart show vtbd0<br />
gpart add -t freebsd-swap -a 4k vtbd0<br />
gpart show vtbd0<br />
swapon /dev/vtbd0p3<br />
<br />
df -h<br />
reboot<br />
<br />
vi /etc/rc.conf<br />
hostname<br />
IP Addr<br />
passwd root<br />
passwd user<br />
clear history<br />
cp /dev/null /var/log/utx.log<br />
cp /dev/null ~root/.history<br />
history -c<br />
<br />
</pre><br />
<br />
= FreeBSD Jails =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Making new customer jail 4.x ==<br />
<br />
1. run [[#js|js]] to figure out which partition, IP to put it on, find an unused vn, and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vnX hostname shorthost quad/safe# ipfw# email [gb disk, default 4]</tt><br />
<br />
jail14# /tmp/jailmake 69.55.226.152 /mnt/data2/69.55.226.152-col00182 vn23 negev.cerastes.org negev 4 182 cerastes@cerastes.org<br />
<br />
<br />
== Making new customer jail 6.x ==<br />
<br />
1. run [[#js|js]] to figure out which gvinum vols are free (and which mnt (disk) each gvinum goes with/to), IP to put it on and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vN[,vN] hostname shorthost quad/safe# ipfw# email</tt><br />
<br />
jail19# jailmake 69.55.236.17 /mnt/data1/69.55.236.17-col01574 v10 uncle-leo.commadev.com uncle-leo 1 1574 lee@commadev.com,lee@gmail.com<br />
<br />
<br />
== Changing an ip for freebsd VPS ==<br />
<br />
*stopjail (hostname)<br />
*on systems using vnfile or mdfile: rename vnfile/mdfile, using new ip<br />
*rename dir using new ip<br />
*Edit quad (make sure to update all lines: directory, vn/mdfile)<br />
*buildsafe (not necessary on systems having only quad1)<br />
*if you're adding an ip not already on the host machine: <tt>ipadd x.x.x.x</tt><br />
*startjail (hostname)<br />
if backups: <tt>mvbackups</tt> ...<br />
*Edit dir/ip in mgmt<br />
*if any rules, update any firewall rules to use new ip (careful not to make sure there arent multiple rules per ipfw#- search by colo#)<br />
*(if customer asks and has domains) update their domains on ns1c<br />
*optional: update ipfw counters on the host server<br />
<br />
<br />
== Rename a gconcat vol ==<br />
<br />
*stop jail, unmount everything<br />
*gconcat stop vol<br />
*gconcat clear /dev/gvinum/vN<br />
*gconcat clear /dev/gvinum/vN+1<br />
*gconcat label -v newname /dev/gvinum/vN /dev/gvinum/vN+1<br />
*bsdlabel /dev/concat/newname<br />
(make sure a partition is not fstype unused, if so change to: 4.2BSD)<br />
<br />
<br />
== Remaking a VPS (on same jail) ==<br />
<br />
# take him down (stopjail)<br />
# make note of host, vn/gvin/md, ip<br />
# remove from quad/safe<br />
# move vnfile (rename)<br />
# remove dir (jailmake will make a new one)<br />
# remove ipfw counts<br />
# jailmake<br />
# update backup script<br />
# update db with new dir/ip (use the “move” link, mark old as stopped today)<br />
# update firewall if changed ip<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
<br />
== Adding disk to a 4.x jail ==<br />
<br />
NOTE: if you unmount before vnconfig, re-vnconfig then unmount then vnunconfig<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them.<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to dump to a different partition and create the new vn on the current partition. If no space to make make dump, then move old vn’s to backup or worse, dump to backup.<br />
<br />
<br />
2. <tt>g <customerID></tt> to get the info<br />
<br />
<br />
3. make vnfile, newfs and mount. USE CAUTION not to overwrite existing vn. <br><br />
Ex: <tt>vnconfig –T -S 9g -s labels -c /dev/vn22 /mnt/data2/65.214.160.117-col00496<br><br />
disklabel -r -w vn30 auto<br><br />
newfs /dev/vn30c<br><br />
mkdir /mnt/data2/65.214.160.117-col00496-DIR<br><br />
mount /dev/vn30c /mnt/data2/69.55.237.26-col00241-DIR<br><br />
cd /mnt/data2/65.214.160.117-col00496-DIR</tt><br />
<br />
<br />
4. <tt>jailkill <hostname></tt><br />
<br />
<br />
5. <tt>dump -0a -f - /dev/vn1 | restore -r -f -</tt> <br />
<br />
<br />
6. <tt>rm restoresymtable</tt><br />
<br />
<br />
7. unmount and unvnconfig old system:<br><br />
<tt>umount /dev/vnNNc <br><br />
vnconfig –u /dev/vnNN</tt><br />
<br />
<br />
8. move vnfile. Ex: <tt>mv /mnt/data1/69.55.237.26-col00241 /mnt/data1/old-col00241-vnfile-20110312-noarchive</tt><br />
<br />
<br />
9. remove old dir: <tt>rmdir /mnt/data1/69.55.237.26-col00241-DIR</tt><br />
<br />
<br />
10. edit quad/safe to point to new location, run <tt>[[#buildsafe|buildsafe]]</tt><br />
<br />
<br />
11. start jail: <tt>[[#startjail|startjail]] <hostname></tt><br />
<br />
<br />
12. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
13. mv backups if necessary<br />
<br />
WARNING: if you restore to unmounted vn then you’re actually restoring to /mnt/data1<br />
you can tar and do above to recover<br />
<br />
NOTE: you can’t move a customer to a system that has a base sys that is diff from the one they came from (BSD 4.8 to 4.5 = won’t work)<br />
<br />
NOTE: we don’t mount procfs's to anyone’s system by default.<br />
<br />
NOTE: mount_nulls: <br />
also goofy mount_nulls seen in df -k on jail2<br><br />
also needs to be done in safe and quad<br><br />
if he wants another, use safe/quad as example and add it to file. also exec command at prompt to add mount (no reboot, remount or rejail necessary)<br />
<br />
== Moving customer to another jail machine ==<br />
<br />
Systems should only be moved between similarly-versioned jails. If you cannot get a match, you should go to freebsd.org and see what the differences were between the version you're moving from and the new version. Many of the updates will be to drivers and kernels - these don't affect the customer. Perhaps they updated traceroute or a library...you should copy the new/changed files into the VPS/jail on the new host. An alternate method for moving to a different version would be to buildworld to bring their full distribution up to date. Optional steps for this kind of upgrade are included below.<br />
<br />
1. <tt>g <customerID></tt><br />
<br />
<br />
2. <tt>[[#jailkill|jailkill]] <hostname></tt><br />
<br />
<br />
3. create new device on target system<br><br />
4.x:<br><br />
<pre>vnconfig –T –S 4g –s labels –c /dev/vn1 /mnt/data1/69.55.22x.x-col00XXX<br />
disklabel –r –w vn1 auto<br />
newfs /dev/vn1c</pre><br />
<br />
6.x:<br><br />
<pre>bsdlabel -r -w /dev/gvinum/v1<br />
newfs /dev/gvinum/v1a<br />
or <br />
gconcat label v1-v3 /dev/gvinum/v1 /dev/gvinum/v2 /dev/gvinum/v3<br />
bsdlabel -r -w /dev/concat/v1-v3<br />
newfs /dev/concat/v1-v3a</pre><br />
<br />
7.x+:<br><br />
Run jailmakeempty then skip step 4 & 7 below<br />
<br />
<br />
4. make and mount the dir on new system. <br><br />
Ex: <pre>mkdir /mnt/data2/69.55.230.3-col00123-DIR<br />
chmod 755 /mnt/data2/69.55.230.3-col00123-DIR<br />
mount [device] /mnt/data2/69.55.230.3-col00123-DIR</pre><br />
<br />
<br />
5. [[#stopjail|stopjail]] <hostname> 1<br />
<br />
<br />
6. dump fs to new system. ex: <tt>[[#dumpremoterestore|dumpremoterestore]] /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR</tt> (make sure you can ssh as root on remote machine) <br />
<br />
<br />
6a. OPTIONAL BUILDWORLD:<br><br />
cd /usr/src<br />
make world DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
cd etc<br />
make distribution DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
(you may have to rm an openssh file and re-make dist)<br />
<br />
rm -rf /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/periodic/daily/400.status-disks<br />
vi /etc/periodic/security/100.chksetuid<br />
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt><br><br />
with: <tt>MP='/' (use single quotes)</tt><br />
<br />
Ask user if they want ports overwritten with current, if yes:<br />
cp -r /usr/ports /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/usr<br />
<br />
on source: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d .<br />
(copy in all info)<br />
<br />
on target: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d . <br />
(paste all info)<br />
<br />
on source: <br />
cat /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group<br />
(copy in all info)<br />
<br />
on target:<br />
cat > /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group <br />
(paste all info)<br />
<br />
<br />
7. edit quad on source system copy over entries for the jail to the target system <br />
take care that the vn/gvinum/gconcat/md devices on the target system are’t in use and that the /mnt/dataN path doesn’t need changing<br />
<br />
<br />
8. run [[#buildsafe|buildsafe]] (if this is <=6.x) on target system (copies changes made to quad into safe file)<br />
<br />
<br />
9. remove ip from source system. Ex: <tt>ipdel 69.55.230.3</tt><br />
<br />
<br />
10. add ip to target system. Ex: <tt>ipadd 69.55.230.3</tt><br />
<br />
<br />
11. start new system, with [[#startjail|startjail]] or manually, pasting entries found from running <tt>g <customerID></tt> on the new system<br />
<br />
<br />
12. run <tt>[[#canceljail|canceljail]] col0xxxx</tt> on source system. This should prompt you to remove backups if any existed. Do not let it do this, rather:<br />
<br />
<br />
13. if backups existed move them to the new host via the <tt>[[#mvbackups_.28freebsd.29|mvbackups]] script</tt><br />
<br />
<br />
14. edit quad on source system, edit comment to reflect a move rather than a cancel, ex: <tt># moved to jail2 col00241</tt><br />
<br />
<br />
15. edit mgmt to reflect new host and dir for new system<br />
<br />
<br />
16. optional add ipfw rules on new system. Ex:<br />
<tt>ipfw add 01231 count ip from 69.55.230.2 to any<br />
ipfw add 01232 count ip from any to 69.55.230.2</tt><br />
<br />
== Increasing inodes for a VPS ==<br />
<br />
when doing the newfs: <br />
newfs -i 4096 ...<br />
<br />
<br />
== Jail services unable to contact services within same jail ==<br />
<br />
This is due to a messed up routing table (only seen it on jail2 before). <br />
<br />
route delete 69.55.228.65/32 route add -net 69.55.228.65/32 -iface fxp0 -nostatic -cloning route add 69.55.228.65 -iface lo0<br />
<br />
<br />
== /dev/null permission resets == <br />
<br />
Applies to 4.x jails only<br />
<br />
For some reason that I do not understand at all, the /dev/null node in customer jails, after the system is restarted, often reverts permissions to 0600 ... which is bad, because non-root processes cannot redirect to /dev/null anymore. In fact, a fair number of server daemons will complain and/or fail if this is the case.<br />
<br />
So, after a restart, after all systems have restarted, run postboot, which does something like:<br />
<br />
for f in `df -k | grep /dev/vn | awk '{print $6}'` ; do chmod 0666 $f/dev/null ; done<br />
<br />
Later, after the system is up, if anyone ever complains about their /dev/null permissions for any reason ... really if anyone mentions<br />
/dev/null in any way, tell them to run:<br />
<br />
chmod 0666 /dev/null<br />
<br />
We have ultimately fixed this by adding chmods to the quad/safe's<br />
<br />
<br />
== mknod in a jail == <br />
<br />
For a customer who wants to run a chroot name server (named) in a jail.<br />
<br />
<pre><br />
cd /mnt/data1/<user directory>/var/named<br />
mknod -c 1 1 null<br />
mknod -c 2 3 random<br />
</pre><br />
<br />
== Postfix Problems ==<br />
<br />
This hasn't happened in a long time, probably no longer valid with modern OS's<br />
<br />
Postfix is an alternate MTA - people replace sendmail with it in much the same way that many people also replace sendmail with qmail. The problem is that if you install postfix inside of a jail, by default it will not work properly - the user will see error messages like this:<br />
<br />
incoming mail generates errors like this:<br />
<br />
mail_queue_enter: create file incoming/298266.55902: File too large<br />
<br />
and further, sending mail generates errors like this:<br />
<br />
postdrop: warning: mail_queue_enter: create file maildrop/627930.56676: File too large<br />
<br />
This is very easy to solve. In fact, the FreeBSD welcome email that is sent out by `jailmake` now contains this block of text:<br />
<br />
- if you plan on installing postfix, email us first and ask for a necessary patch.<br />
<br />
So, anytime you see a support email that says anything about postfix not working, paste that line from the welcome email into your response to them, and right away, forward the postfix patch to them in an email.<br />
<br />
Here are the instructions:<br />
<br />
Simply copy the patch into /usr/ports/mail/postfix/files, rename the file to ‘patch-file_limit.c’ and then cd to /usr/ports/mail/postfix and run `make install`.<br />
<br />
The postfix patch is the very first email in the support email box, and we never delete or save it because we always want it there to forward to people. The instructions in the email are very clear, and it solves their problem 100% of the time.<br />
<br />
<br />
== Problems with `find` ==<br />
<br />
There was a bug in the FreeBSD vn-filesystem code. The bug causes the system to crash or hang when someone runs the `find` command inside their vn-backed filesystem (and even though the bug may have been fixed, due to the i/o overhead, we still don't like find).<br />
<br />
Now, this does not always happen - however, any jail machine with more than 20 systems on it will crash every night if every system on it runs the daily periodic script out of their crontab, and updates their locate database with the find command.<br />
<br />
(the locate database is a small, FreeBSD specific database that is populated nightly from a cron job, and is then used to provide fast answers to the `locate` command)<br />
<br />
In normal systems (32-40 jails on them), all that has to be done is make sure nobody runs the daily periodic. This means that the file /etc/crontab, instead of looking like this:<br />
<br />
<pre># do daily/weekly/monthly maintenance<br />
1 3 * * * root periodic daily<br />
15 4 * * 6 root periodic weekly<br />
30 5 1 * * root periodic monthly</pre><br />
<br />
Needs to look like this:<br />
<br />
<pre># DO NOT UNCOMMENT THESE - contact support@johncompanies.com for details<br />
#1 3 * * * root periodic daily<br />
#15 4 * * 6 root periodic weekly<br />
#30 5 1 * * root periodic monthly<br />
## DO NOT UNCOMMENT THESE ^^^^^^^^</pre><br />
<br />
and the problem will generally not occur at all. New filesystem images are always altered in this way, so all systems are set like this.<br />
<br />
However, if you have enough systems running, even casual uses of find, outside of the periodic scripts, can cause the system to crash. We are seeing this on jail13, which crashes about once per week.<br />
<br />
There is not yet any resolution to this problem.<br />
<br />
So, if you have a freebsd system crash, see if anyone has uncommented their daily periodic line - you can see them for everyone by running:<br />
<br />
/bin/sh<br />
<br />
and then running:<br />
<br />
<pre>for f in `df -k | grep vn | awk '{print $6}'` ; do echo `cat $f/etc/crontab | grep "periodic daily"` $f ; done | more</pre><br />
<br />
(all one line, one command)<br />
<br />
And then edit those /etc/crontabs and comment them out again.<br />
<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
* if you are getting disk full messages for a BSD customer, it's fairly safe to clear out their /usr/ports/distfiles dir<br />
* 4.x: ps and top an only be run by root in these jails. Done on purpose:<br />
As for `ps` and `top` - non-root users can run them, just not successfully because we have locked the permissions on /dev/mem and /dev/kmem to be root-readable only. That is why non-root users cannot successfully run `ps` and `top`.<br />
* user quotas do not work on freebsd jails - you cannot set up quotas at all, and that's that.<br />
* You cannot inject a process into a 4.x jail, only HUP running processes. 6.x and onward you can with jexec<br />
* jails see base machine’s uptime/load when running top/w<br />
* if someone is unable to get in- cant ping, etc- see if they were blocked by castle (DoS), see if their ip is on the system (post reboot it was lost cause wasn’t in the rc.conf). preboot should catch that.<br />
* in FreeBSD you cant su to root unless you belong to wheel group – so if you remove your acct and setup a new one, we have to add it to wheel group (add to /etc/group)<br />
* Dmesg from underlying sys is seen in customers dmesg on jail<br />
* Popper process going crazy @ 40% for 10min = someone who leaves mail on server<br />
* Don’t force umounts on 4.x jails – it crashes the machine, generally ok on newer machines (running md)<br />
* Good book for admin http://search.barnesandnoble.com/booksearch/isbnInquiry.asp?userid=t824VyRAYz&isbn=0596005164&itm=2<br />
* Self-sign ssl cert http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#selfcert<br />
* conversation with Glenn about semaphores and pgsql<br />
<pre>SDBoody: hey, these are valid amounts/figures/increments right:<br />
kern.ipc.semmni=1280<br />
kern.ipc.semmns=1280<br />
<br />
gr8feen: probably... I always forget exactly what those are, so I usually have to look them up<br />
SDBoody: semaphores<br />
SDBoody: i took the current 1024 and added 256 to them<br />
SDBoody: need more for pgsql<br />
gr8feen: I meant the mni and mns parts...<br />
gr8feen: some of those are not ones you just want to add to... hang on a sec and I'll look them up..<br />
gr8feen: what's semmsl set to?<br />
SDBoody: kern.ipc.semmsl: 1024<br />
<br />
SDBoody: kern.ipc.msgseg: 2048 <br />
kern.ipc.msgssz: 8 <br />
kern.ipc.msgtql: 40 <br />
kern.ipc.msgmnb: 2048 <br />
kern.ipc.msgmni: 40 <br />
kern.ipc.msgmax: 16384 <br />
kern.ipc.semaem: 16384 <br />
kern.ipc.semvmx: 65534 <br />
kern.ipc.semusz: 152 <br />
kern.ipc.semume: 10 <br />
kern.ipc.semopm: 100 <br />
kern.ipc.semmsl: 1024 <br />
kern.ipc.semmnu: 512 <br />
kern.ipc.semmns: 1024 <br />
kern.ipc.semmni: 1024 <br />
kern.ipc.semmap: 768 <br />
kern.ipc.shm_allow_removed: 0 <br />
kern.ipc.shm_use_phys: 1 <br />
kern.ipc.shmall: 262144 <br />
kern.ipc.shmseg: 256 <br />
kern.ipc.shmmni: 784 <br />
kern.ipc.shmmin: 1 <br />
kern.ipc.shmmax: 536870912 <br />
kern.ipc.maxsockets: 25600 <br />
<br />
gr8feen: ok...msl is max per id, mni is max ids, mns is max number of semaphores... so you probably want something like mns = mni * msl<br />
gr8feen: which one did you run out of?<br />
SDBoody: not sure how to tell- ipcs shows the sems in use add up to 1024<br />
SDBoody: there are 59 entries<br />
gr8feen: I'm assuming you tried to start postgres and it failed?<br />
SDBoody: yes<br />
gr8feen: it should have logged why, somewhere..<br />
gr8feen: if I recall, it'll tell you which one it ran out of<br />
SDBoody: > DETAIL: Failed system call was semget(1, 17, 03600).<br />
<br />
gr8feen: so it wanted an id with 17 semaphores...I'd start by making mns = 17*mni and leave mni and mnl set to what they are now and see what it does<br />
SDBoody: i think mni is plenty high<br />
SDBoody: ok, more reasonable 17408<br />
gr8feen: yeah...just change that one and see how it goes..<br />
SDBoody: and leave mni alone at 1024?<br />
gr8feen: yeah...mni id the max number of ids...but if it's trying to get something like 17 per id, your going to hit mns before you hit anything else<br />
SDBoody: right, but doesn't hurt to have it that high (assuming)<br />
<br />
gr8feen: not really... I think those get allocated out of ram that you cant page out, but it's still such a small amount that it really doesn't matter<br />
SDBoody: looks like that worked, thx!<br />
gr8feen: cool<br />
<br />
see semaphores: ipcs -a -s</pre><br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
== Making new customer VE (vm) ==<br />
<br />
This applies only to new virts >= 4.x<br />
<br />
grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.<br />
<br />
1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
also, ping the IP address.<br />
<br />
2. put CT on whichever partition has more space<br />
df -h<br />
<br />
3. vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> ) <br />
vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@johncompanies.com centos-6-x86_64 LM<br />
<br />
4. copy veid, dir, ip and password to pending customer screen. activate customer<br />
<br />
== Making new customer VE (vemakexxx) ==<br />
<br />
This applies to older virts with old templates. This should probably not be used at all anymore.<br />
<br />
<br />
1. look thru hist for ip<br />
<br />
2. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
<br />
3. put ve on whichever partition has more space<br />
vemakerh9 <veid> <ip> <hostname> <mount> <email> [gb disk]; <256|384|512> <veid><br />
vemakerh9 866 69.55.226.109 ngentu.com /vz1 ayo@ngantu.com,asd@asd.com 5; 256 866<br />
<br />
4. copy (veid), dir, and ip to pending customer screen (pass set to p455agfa)<br />
<br />
<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrateonline/migrate failures (migrate manually) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
You may also see migration failing due to quota issues.<br />
<br />
You can try to resolve by copying any quota file into the file you need:<br />
<br />
cp /var/vzquota/quota.1 /var/vzquota/quota.xxx<br />
<br />
If it complains about quota running you should then be able to stop it<br />
<br />
vzquota off xxxx<br />
<br />
If all else fails, migrate to a new VEID<br />
i.e. 1234 becomes 12341<br />
<br />
<br />
If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
'''Note:''' don't forget to fix /usr/local/sbin/backup.config for new disk location<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
=== Cant change file /etc/sysconfig/network ===<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
=== What template is user using ===<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
=== Edits needed to run java ===<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
=== Set time zone ===<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
=== Failing shm_open calls ===<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
=== Debian can't get on the network ===<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
=== CentOS can't get on the network ===<br />
<br />
When CentOS system was running, but can't get on the network, they probably updated iproute via yum.<br />
<br />
Copy an older version of the iproute rpm to their VPS.<br />
<br />
<pre>cp -p /root/iproute/iproute-2.6.32-23.el6.<i686>.rpm /vz2/private/<1527>/fs/root</pre><br />
<br />
Enter their ve, install rpm, restart network, and prevent yum from updating iproute.<br />
<br />
<pre><br />
ve <1527><br />
rpm -i --force iproute-2.6.32-23.el6.<i686>.rpm<br />
rpm -q iproute<br />
rpm -e <new iproute rpm><br />
service network restart<br />
echo "exclude=iproute" >>/etc/yum/ip-exclude<br />
echo "include=/etc/yum/ip-exclude" >>/etc/yum.conf<br />
</pre><br />
<br />
<br />
<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
----<br />
<br />
* uptime/load that vz users see isn’t what we see- it's their own load<br />
* Vz overwrites resolve.conf on every boot to what’s in ve config file. Also overwrites hostname<br />
* Quotas in vz: set Gid_limit in vz config file and restart ve. Errors leading up to this will be out of files but no inode or space issues. set to 2000 in conf, and restart vz<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
To create a new Linux VPS<br />
<br />
* get an IP address from mgmt screen<br />
* run df -h to figure out which disk has enough space<br />
* run vm with no arguments to get syntax and exact template name<br />
* use col0xxxx for hostname if they don’t give you a hostname<br />
* copy over dir, veid, ip and password to pending customer screen<br />
<br />
vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> )<br />
<br />
== disable ==<br />
To disable a customers Virtuozzo VPS server.<br />
<br />
* stop the server<br />
vzctl stop <veid><br />
* prevent restart on server reboot and user restart via Control Panel<br />
vzctl set <veid> --disabled=yes --offline_management=no --save<br />
<br />
== cancelve ==<br />
When a customer cancels all service.<br />
<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
<br />
display the contents of the vz config file for the <veid><br />
<br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
<br />
List all virtual environments running on this machine.<br />
<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
<br />
show processes running under this <veid>.<br />
<br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
<br />
Restart a <veid>.<br />
<br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
<br />
enter a <veid> as root.<br />
<br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
<br />
execute a single command in the <veid> environment.<br />
<br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem VEID <256|512|768|1024|1280|1536|2048> [burst GB]<br />
adjusts the memory resources for the VE. If this is on VZ ver >4, you will also be prompted to provide a burst ram amount. Sample usage:<br />
<br />
<pre># setmem 21671 1024 4<br />
/usr/sbin/vzctl set 21671 --ram 1024m --swap 3072m --save<br />
</pre><br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>75.80.128.147https://wiki.jcihosting.com/index.php?title=VPS_Management&diff=2228VPS Management2020-04-30T20:27:03Z<p>75.80.128.147: /* Ganeti Install a Linux VPS */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
== Moving a server to a different account in the management system ==<br />
<br />
This is a tough one. You gotta touch a bunch of tables in the mysql database.<br />
<pre><br />
systems<br />
subs<br />
ipmap<br />
</pre><br />
<br />
basically, get the sysid and go off that:<br />
<br />
<pre><br />
update systems set cid='newcid' where sysid=xxxxx<br />
update subs set cid='newcid' where sysid=xxxxx<br />
update ipmap set cid='newcid' where sysid=xxxxx<br />
</pre><br />
<br />
You then need to rename the server disk space to change the old ID to the new one.<br />
<br />
For a jail -<br />
<pre><br />
stopjail <hostname><br />
mv 69.55.2XX.XXX-col0<oldid> 69.55.2XX.XXX-col0<newid><br />
mv 69.55.2XX.XXX-col0<oldid>-DIR 69.55.2XX.XXX-col0<newid>-DIR<br />
vi /usr/local/jail/bin/quad1 [Update the ID in the quad file]<br />
startjail <hostname><br />
</pre><br />
<br />
For a virt -<br />
You will have to <br />
* stop the virt <br />
* rename the directories <br />
* modify the config file /vzconf/<veid>.conf <br />
* rename the virt conf file <br />
* restart the new virt.<br />
<br />
that ought to do it..<br />
<br />
<br />
== Re-enable a customer account in the management system ==<br />
<br />
For example to re-instate col03438, In mysql:<br />
<br />
<pre><br />
show tables ;<br />
select * from customers where cid='col03438' ;<br />
update customers set status=1 where cid='col03438' ;<br />
select * from systems where cid='col03438' ;<br />
update systems set stop_date=NULL where cid='col03438' ;<br />
update systems set cancel_date=NULL where cid='col03438' ;<br />
update systems set status=1 where cid='col03438' ;<br />
select * from subs where cid='col03438' ;<br />
update subs set stop_date=NULL where cid='col03438' ;<br />
update subs set status=1 where cid='col03438' <br />
<br />
</pre><br />
<br />
= Ganeti VPS =<br />
<br />
This is how to create a VPS on the Lamphost ganeti cluster.<br />
I have created scripts for creating a FreeBSD VPS and one for<br />
creating a Linux VPS.<br />
<br />
== Ganeti Install a Linux VPS ==<br />
<br />
To create a linux VPS use<br />
<pre><br />
/root/new <name> <disk size> <memory> <IP Addr> [<OS>]<br />
<br />
( for example /root/new vps.example.com 30 2 69.55.238.10 centos-8 )<br />
</pre><br />
<br />
You need to customize the install by adding a "jci" account and a customer<br />
account (col0XXXX) as well. <br />
<br />
<br />
== Ganeti Install a FreeBSD VPS ==<br />
<br />
To create a FreeBSD VPS, we have to do a full install. This will <br />
create a new VPS.<br />
<br />
<pre><br />
/root/newfreebsd12 <name> <disk size> <memory> <IP Addr> [<OS>]<br />
</pre><br />
<br />
You will then have to do the install via VNC.<br />
<br />
<br><br />
You need to customize the install by adding a "jci" account and a customer<br />
account (col0XXXX) as well. <br />
<br />
<br> <br />
You will need to install sudo.<br />
<br />
<pre> <br />
pkg install sudo # install sudo package<br />
visudo # to add wheel account to sudoers<br />
vi /etc/group # add jci and user acct to wheel<br />
</pre><br />
<br />
<br><br />
You will also have to set up a console in /etc/ttys.<br />
<pre><br />
vi /etc/ttys<br />
</pre><br />
<br />
== Ganeti restart a VPS ==<br />
<br />
To shutdown a ganeti VPS<br />
<br />
<pre><br />
gnt-instance reboot <VPSname><br />
</pre><br />
<br />
<br />
== Ganeti shutdown a VPS ==<br />
<br />
To shutdown a ganeti VPS<br />
<br />
<pre><br />
gnt-instance shutdown <VPSname><br />
</pre><br />
<br />
To permanently delete a ganeti VPS<br />
<br />
<pre><br />
gnt-instance remove <VPSname><br />
</pre><br />
<br />
== Ganeti Re-Intsall a VPS ==<br />
<br />
Re-installing the OS will destroy all data on the VPS.<br />
Login to the Ganeti master node (currently gn1).<br />
<br />
To reinstall the OS on a VPS the VPS must be shutdown.<br />
<pre><br />
gnt-instance shutdown <VPSname><br />
</pre><br />
<br />
Do the reinstall.<br />
<pre><br />
gnt-instance reinstall <VPSname><br />
</pre><br />
<br />
Restart the VPS.<br />
<pre><br />
gnt-instance startup <VPSname><br />
</pre><br />
<br />
Now you need to do the customization for the VPS. This is<br />
basically the same as the Dedicated server customization.<br />
<br />
https://69.55.229.8/index.php/New_Signups#Dedicated_Server_Install_Checklist<br />
<br />
= FreeBSD 10.1 VPS (jail5 and jail6) =<br />
<br />
This is information on how to manage the FreeBSD 10.1 VPS servers on Jail5.<br />
<br />
<br />
== Making new customer VPS ==<br />
<pre><br />
virtmake IP[,IP] <CID> hostname ipfw# email {FSP|FB|FS|FM|FP} [size in GB]<br />
</pre><br />
<br />
Note: <br />
For now, CID must be unique in the last 2 digits of the CID.<br />
<br />
Must do a "vm start <CID>" after creating a VPS.<br />
<br />
== Starting VPS ==<br />
To start all vms on the system<br />
<pre><br />
vm start<br />
</pre><br />
<br />
To start a single VPS <br />
<pre><br />
vm start <CID><br />
</pre><br />
<br />
== List running VPS ==<br />
<pre><br />
vm status<br />
</pre><br />
<br />
== Attach to a user VPS console ==<br />
<pre><br />
vm attach <CID><br />
</pre><br />
<br />
<br />
To detach from a console, type tilde control-d (~ ctl-d).<br />
<br />
== Stop a VPS ==<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
== Restart a VPS ==<br />
<pre><br />
vm restart <CID><br />
</pre><br />
<br />
<br />
== Modify a VPS ==<br />
<br />
To modify a VPS config, stop the VPS<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
Mount the VPS data file<br />
<pre><br />
vm mount <CID><br />
<br />
cd /data/vm/private/<CID>/mount<br />
</pre><br />
<br />
Modify the user files necessary.<br />
<br />
Restart the VPS.<br />
<br />
<pre><br />
vm umount <CID><br />
<br />
vm start <CID><br />
</pre><br />
<br />
== Notes on a new FBSD 10.x VPS (jail6) ==<br />
<br />
<pre><br />
#! /bin/sh<br />
#<br />
# new <colo#> <disk><br />
<br />
<br />
memory=1024<br />
<br />
x="$1"<br />
<br />
disk="$2"<br />
if [ "y" == "y$disk" ] ; then<br />
disk=60<br />
fi<br />
<br />
disk2=`expr $disk - 4`<br />
<br />
<br />
vm image provision 04dcdf96-d843-11e6-b131-001e67fcc64f col0$x<br />
# vm configure col0$x<br />
sed -e "s/memory=256M/memory=${memory}M/" /vms/col01114/col01114.conf >/vms/col0$x/col0$x.conf<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
# vm start col0$x<br />
<br />
<br />
exit<br />
<br />
vm console col0$x<br />
<br />
#<br />
# to increase the customer disk<br />
#<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
<br />
## on the customer VPS<br />
<br />
gpart recover vtbd0<br />
gpart show vtbd0<br />
swapoff /dev/vtbd0p3<br />
gpart delete -i 3 vtbd0<br />
gpart show vtbd0<br />
sysctl kern.geom.debugflags=16<br />
gpart resize -i 2 -a 4k -s $disk2\G vtbd0<br />
<br />
growfs /dev/vtbd0p2<br />
<br />
gpart show vtbd0<br />
gpart add -t freebsd-swap -a 4k vtbd0<br />
gpart show vtbd0<br />
swapon /dev/vtbd0p3<br />
<br />
df -h<br />
reboot<br />
<br />
vi /etc/rc.conf<br />
hostname<br />
IP Addr<br />
passwd root<br />
passwd user<br />
clear history<br />
cp /dev/null /var/log/utx.log<br />
cp /dev/null ~root/.history<br />
history -c<br />
<br />
</pre><br />
<br />
= FreeBSD Jails =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Making new customer jail 4.x ==<br />
<br />
1. run [[#js|js]] to figure out which partition, IP to put it on, find an unused vn, and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vnX hostname shorthost quad/safe# ipfw# email [gb disk, default 4]</tt><br />
<br />
jail14# /tmp/jailmake 69.55.226.152 /mnt/data2/69.55.226.152-col00182 vn23 negev.cerastes.org negev 4 182 cerastes@cerastes.org<br />
<br />
<br />
== Making new customer jail 6.x ==<br />
<br />
1. run [[#js|js]] to figure out which gvinum vols are free (and which mnt (disk) each gvinum goes with/to), IP to put it on and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vN[,vN] hostname shorthost quad/safe# ipfw# email</tt><br />
<br />
jail19# jailmake 69.55.236.17 /mnt/data1/69.55.236.17-col01574 v10 uncle-leo.commadev.com uncle-leo 1 1574 lee@commadev.com,lee@gmail.com<br />
<br />
<br />
== Changing an ip for freebsd VPS ==<br />
<br />
*stopjail (hostname)<br />
*on systems using vnfile or mdfile: rename vnfile/mdfile, using new ip<br />
*rename dir using new ip<br />
*Edit quad (make sure to update all lines: directory, vn/mdfile)<br />
*buildsafe (not necessary on systems having only quad1)<br />
*if you're adding an ip not already on the host machine: <tt>ipadd x.x.x.x</tt><br />
*startjail (hostname)<br />
if backups: <tt>mvbackups</tt> ...<br />
*Edit dir/ip in mgmt<br />
*if any rules, update any firewall rules to use new ip (careful not to make sure there arent multiple rules per ipfw#- search by colo#)<br />
*(if customer asks and has domains) update their domains on ns1c<br />
*optional: update ipfw counters on the host server<br />
<br />
<br />
== Rename a gconcat vol ==<br />
<br />
*stop jail, unmount everything<br />
*gconcat stop vol<br />
*gconcat clear /dev/gvinum/vN<br />
*gconcat clear /dev/gvinum/vN+1<br />
*gconcat label -v newname /dev/gvinum/vN /dev/gvinum/vN+1<br />
*bsdlabel /dev/concat/newname<br />
(make sure a partition is not fstype unused, if so change to: 4.2BSD)<br />
<br />
<br />
== Remaking a VPS (on same jail) ==<br />
<br />
# take him down (stopjail)<br />
# make note of host, vn/gvin/md, ip<br />
# remove from quad/safe<br />
# move vnfile (rename)<br />
# remove dir (jailmake will make a new one)<br />
# remove ipfw counts<br />
# jailmake<br />
# update backup script<br />
# update db with new dir/ip (use the “move” link, mark old as stopped today)<br />
# update firewall if changed ip<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
<br />
== Adding disk to a 4.x jail ==<br />
<br />
NOTE: if you unmount before vnconfig, re-vnconfig then unmount then vnunconfig<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them.<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to dump to a different partition and create the new vn on the current partition. If no space to make make dump, then move old vn’s to backup or worse, dump to backup.<br />
<br />
<br />
2. <tt>g <customerID></tt> to get the info<br />
<br />
<br />
3. make vnfile, newfs and mount. USE CAUTION not to overwrite existing vn. <br><br />
Ex: <tt>vnconfig –T -S 9g -s labels -c /dev/vn22 /mnt/data2/65.214.160.117-col00496<br><br />
disklabel -r -w vn30 auto<br><br />
newfs /dev/vn30c<br><br />
mkdir /mnt/data2/65.214.160.117-col00496-DIR<br><br />
mount /dev/vn30c /mnt/data2/69.55.237.26-col00241-DIR<br><br />
cd /mnt/data2/65.214.160.117-col00496-DIR</tt><br />
<br />
<br />
4. <tt>jailkill <hostname></tt><br />
<br />
<br />
5. <tt>dump -0a -f - /dev/vn1 | restore -r -f -</tt> <br />
<br />
<br />
6. <tt>rm restoresymtable</tt><br />
<br />
<br />
7. unmount and unvnconfig old system:<br><br />
<tt>umount /dev/vnNNc <br><br />
vnconfig –u /dev/vnNN</tt><br />
<br />
<br />
8. move vnfile. Ex: <tt>mv /mnt/data1/69.55.237.26-col00241 /mnt/data1/old-col00241-vnfile-20110312-noarchive</tt><br />
<br />
<br />
9. remove old dir: <tt>rmdir /mnt/data1/69.55.237.26-col00241-DIR</tt><br />
<br />
<br />
10. edit quad/safe to point to new location, run <tt>[[#buildsafe|buildsafe]]</tt><br />
<br />
<br />
11. start jail: <tt>[[#startjail|startjail]] <hostname></tt><br />
<br />
<br />
12. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
13. mv backups if necessary<br />
<br />
WARNING: if you restore to unmounted vn then you’re actually restoring to /mnt/data1<br />
you can tar and do above to recover<br />
<br />
NOTE: you can’t move a customer to a system that has a base sys that is diff from the one they came from (BSD 4.8 to 4.5 = won’t work)<br />
<br />
NOTE: we don’t mount procfs's to anyone’s system by default.<br />
<br />
NOTE: mount_nulls: <br />
also goofy mount_nulls seen in df -k on jail2<br><br />
also needs to be done in safe and quad<br><br />
if he wants another, use safe/quad as example and add it to file. also exec command at prompt to add mount (no reboot, remount or rejail necessary)<br />
<br />
== Moving customer to another jail machine ==<br />
<br />
Systems should only be moved between similarly-versioned jails. If you cannot get a match, you should go to freebsd.org and see what the differences were between the version you're moving from and the new version. Many of the updates will be to drivers and kernels - these don't affect the customer. Perhaps they updated traceroute or a library...you should copy the new/changed files into the VPS/jail on the new host. An alternate method for moving to a different version would be to buildworld to bring their full distribution up to date. Optional steps for this kind of upgrade are included below.<br />
<br />
1. <tt>g <customerID></tt><br />
<br />
<br />
2. <tt>[[#jailkill|jailkill]] <hostname></tt><br />
<br />
<br />
3. create new device on target system<br><br />
4.x:<br><br />
<pre>vnconfig –T –S 4g –s labels –c /dev/vn1 /mnt/data1/69.55.22x.x-col00XXX<br />
disklabel –r –w vn1 auto<br />
newfs /dev/vn1c</pre><br />
<br />
6.x:<br><br />
<pre>bsdlabel -r -w /dev/gvinum/v1<br />
newfs /dev/gvinum/v1a<br />
or <br />
gconcat label v1-v3 /dev/gvinum/v1 /dev/gvinum/v2 /dev/gvinum/v3<br />
bsdlabel -r -w /dev/concat/v1-v3<br />
newfs /dev/concat/v1-v3a</pre><br />
<br />
7.x+:<br><br />
Run jailmakeempty then skip step 4 & 7 below<br />
<br />
<br />
4. make and mount the dir on new system. <br><br />
Ex: <pre>mkdir /mnt/data2/69.55.230.3-col00123-DIR<br />
chmod 755 /mnt/data2/69.55.230.3-col00123-DIR<br />
mount [device] /mnt/data2/69.55.230.3-col00123-DIR</pre><br />
<br />
<br />
5. [[#stopjail|stopjail]] <hostname> 1<br />
<br />
<br />
6. dump fs to new system. ex: <tt>[[#dumpremoterestore|dumpremoterestore]] /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR</tt> (make sure you can ssh as root on remote machine) <br />
<br />
<br />
6a. OPTIONAL BUILDWORLD:<br><br />
cd /usr/src<br />
make world DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
cd etc<br />
make distribution DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
(you may have to rm an openssh file and re-make dist)<br />
<br />
rm -rf /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/periodic/daily/400.status-disks<br />
vi /etc/periodic/security/100.chksetuid<br />
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt><br><br />
with: <tt>MP='/' (use single quotes)</tt><br />
<br />
Ask user if they want ports overwritten with current, if yes:<br />
cp -r /usr/ports /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/usr<br />
<br />
on source: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d .<br />
(copy in all info)<br />
<br />
on target: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d . <br />
(paste all info)<br />
<br />
on source: <br />
cat /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group<br />
(copy in all info)<br />
<br />
on target:<br />
cat > /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group <br />
(paste all info)<br />
<br />
<br />
7. edit quad on source system copy over entries for the jail to the target system <br />
take care that the vn/gvinum/gconcat/md devices on the target system are’t in use and that the /mnt/dataN path doesn’t need changing<br />
<br />
<br />
8. run [[#buildsafe|buildsafe]] (if this is <=6.x) on target system (copies changes made to quad into safe file)<br />
<br />
<br />
9. remove ip from source system. Ex: <tt>ipdel 69.55.230.3</tt><br />
<br />
<br />
10. add ip to target system. Ex: <tt>ipadd 69.55.230.3</tt><br />
<br />
<br />
11. start new system, with [[#startjail|startjail]] or manually, pasting entries found from running <tt>g <customerID></tt> on the new system<br />
<br />
<br />
12. run <tt>[[#canceljail|canceljail]] col0xxxx</tt> on source system. This should prompt you to remove backups if any existed. Do not let it do this, rather:<br />
<br />
<br />
13. if backups existed move them to the new host via the <tt>[[#mvbackups_.28freebsd.29|mvbackups]] script</tt><br />
<br />
<br />
14. edit quad on source system, edit comment to reflect a move rather than a cancel, ex: <tt># moved to jail2 col00241</tt><br />
<br />
<br />
15. edit mgmt to reflect new host and dir for new system<br />
<br />
<br />
16. optional add ipfw rules on new system. Ex:<br />
<tt>ipfw add 01231 count ip from 69.55.230.2 to any<br />
ipfw add 01232 count ip from any to 69.55.230.2</tt><br />
<br />
== Increasing inodes for a VPS ==<br />
<br />
when doing the newfs: <br />
newfs -i 4096 ...<br />
<br />
<br />
== Jail services unable to contact services within same jail ==<br />
<br />
This is due to a messed up routing table (only seen it on jail2 before). <br />
<br />
route delete 69.55.228.65/32 route add -net 69.55.228.65/32 -iface fxp0 -nostatic -cloning route add 69.55.228.65 -iface lo0<br />
<br />
<br />
== /dev/null permission resets == <br />
<br />
Applies to 4.x jails only<br />
<br />
For some reason that I do not understand at all, the /dev/null node in customer jails, after the system is restarted, often reverts permissions to 0600 ... which is bad, because non-root processes cannot redirect to /dev/null anymore. In fact, a fair number of server daemons will complain and/or fail if this is the case.<br />
<br />
So, after a restart, after all systems have restarted, run postboot, which does something like:<br />
<br />
for f in `df -k | grep /dev/vn | awk '{print $6}'` ; do chmod 0666 $f/dev/null ; done<br />
<br />
Later, after the system is up, if anyone ever complains about their /dev/null permissions for any reason ... really if anyone mentions<br />
/dev/null in any way, tell them to run:<br />
<br />
chmod 0666 /dev/null<br />
<br />
We have ultimately fixed this by adding chmods to the quad/safe's<br />
<br />
<br />
== mknod in a jail == <br />
<br />
For a customer who wants to run a chroot name server (named) in a jail.<br />
<br />
<pre><br />
cd /mnt/data1/<user directory>/var/named<br />
mknod -c 1 1 null<br />
mknod -c 2 3 random<br />
</pre><br />
<br />
== Postfix Problems ==<br />
<br />
This hasn't happened in a long time, probably no longer valid with modern OS's<br />
<br />
Postfix is an alternate MTA - people replace sendmail with it in much the same way that many people also replace sendmail with qmail. The problem is that if you install postfix inside of a jail, by default it will not work properly - the user will see error messages like this:<br />
<br />
incoming mail generates errors like this:<br />
<br />
mail_queue_enter: create file incoming/298266.55902: File too large<br />
<br />
and further, sending mail generates errors like this:<br />
<br />
postdrop: warning: mail_queue_enter: create file maildrop/627930.56676: File too large<br />
<br />
This is very easy to solve. In fact, the FreeBSD welcome email that is sent out by `jailmake` now contains this block of text:<br />
<br />
- if you plan on installing postfix, email us first and ask for a necessary patch.<br />
<br />
So, anytime you see a support email that says anything about postfix not working, paste that line from the welcome email into your response to them, and right away, forward the postfix patch to them in an email.<br />
<br />
Here are the instructions:<br />
<br />
Simply copy the patch into /usr/ports/mail/postfix/files, rename the file to ‘patch-file_limit.c’ and then cd to /usr/ports/mail/postfix and run `make install`.<br />
<br />
The postfix patch is the very first email in the support email box, and we never delete or save it because we always want it there to forward to people. The instructions in the email are very clear, and it solves their problem 100% of the time.<br />
<br />
<br />
== Problems with `find` ==<br />
<br />
There was a bug in the FreeBSD vn-filesystem code. The bug causes the system to crash or hang when someone runs the `find` command inside their vn-backed filesystem (and even though the bug may have been fixed, due to the i/o overhead, we still don't like find).<br />
<br />
Now, this does not always happen - however, any jail machine with more than 20 systems on it will crash every night if every system on it runs the daily periodic script out of their crontab, and updates their locate database with the find command.<br />
<br />
(the locate database is a small, FreeBSD specific database that is populated nightly from a cron job, and is then used to provide fast answers to the `locate` command)<br />
<br />
In normal systems (32-40 jails on them), all that has to be done is make sure nobody runs the daily periodic. This means that the file /etc/crontab, instead of looking like this:<br />
<br />
<pre># do daily/weekly/monthly maintenance<br />
1 3 * * * root periodic daily<br />
15 4 * * 6 root periodic weekly<br />
30 5 1 * * root periodic monthly</pre><br />
<br />
Needs to look like this:<br />
<br />
<pre># DO NOT UNCOMMENT THESE - contact support@johncompanies.com for details<br />
#1 3 * * * root periodic daily<br />
#15 4 * * 6 root periodic weekly<br />
#30 5 1 * * root periodic monthly<br />
## DO NOT UNCOMMENT THESE ^^^^^^^^</pre><br />
<br />
and the problem will generally not occur at all. New filesystem images are always altered in this way, so all systems are set like this.<br />
<br />
However, if you have enough systems running, even casual uses of find, outside of the periodic scripts, can cause the system to crash. We are seeing this on jail13, which crashes about once per week.<br />
<br />
There is not yet any resolution to this problem.<br />
<br />
So, if you have a freebsd system crash, see if anyone has uncommented their daily periodic line - you can see them for everyone by running:<br />
<br />
/bin/sh<br />
<br />
and then running:<br />
<br />
<pre>for f in `df -k | grep vn | awk '{print $6}'` ; do echo `cat $f/etc/crontab | grep "periodic daily"` $f ; done | more</pre><br />
<br />
(all one line, one command)<br />
<br />
And then edit those /etc/crontabs and comment them out again.<br />
<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
* if you are getting disk full messages for a BSD customer, it's fairly safe to clear out their /usr/ports/distfiles dir<br />
* 4.x: ps and top an only be run by root in these jails. Done on purpose:<br />
As for `ps` and `top` - non-root users can run them, just not successfully because we have locked the permissions on /dev/mem and /dev/kmem to be root-readable only. That is why non-root users cannot successfully run `ps` and `top`.<br />
* user quotas do not work on freebsd jails - you cannot set up quotas at all, and that's that.<br />
* You cannot inject a process into a 4.x jail, only HUP running processes. 6.x and onward you can with jexec<br />
* jails see base machine’s uptime/load when running top/w<br />
* if someone is unable to get in- cant ping, etc- see if they were blocked by castle (DoS), see if their ip is on the system (post reboot it was lost cause wasn’t in the rc.conf). preboot should catch that.<br />
* in FreeBSD you cant su to root unless you belong to wheel group – so if you remove your acct and setup a new one, we have to add it to wheel group (add to /etc/group)<br />
* Dmesg from underlying sys is seen in customers dmesg on jail<br />
* Popper process going crazy @ 40% for 10min = someone who leaves mail on server<br />
* Don’t force umounts on 4.x jails – it crashes the machine, generally ok on newer machines (running md)<br />
* Good book for admin http://search.barnesandnoble.com/booksearch/isbnInquiry.asp?userid=t824VyRAYz&isbn=0596005164&itm=2<br />
* Self-sign ssl cert http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#selfcert<br />
* conversation with Glenn about semaphores and pgsql<br />
<pre>SDBoody: hey, these are valid amounts/figures/increments right:<br />
kern.ipc.semmni=1280<br />
kern.ipc.semmns=1280<br />
<br />
gr8feen: probably... I always forget exactly what those are, so I usually have to look them up<br />
SDBoody: semaphores<br />
SDBoody: i took the current 1024 and added 256 to them<br />
SDBoody: need more for pgsql<br />
gr8feen: I meant the mni and mns parts...<br />
gr8feen: some of those are not ones you just want to add to... hang on a sec and I'll look them up..<br />
gr8feen: what's semmsl set to?<br />
SDBoody: kern.ipc.semmsl: 1024<br />
<br />
SDBoody: kern.ipc.msgseg: 2048 <br />
kern.ipc.msgssz: 8 <br />
kern.ipc.msgtql: 40 <br />
kern.ipc.msgmnb: 2048 <br />
kern.ipc.msgmni: 40 <br />
kern.ipc.msgmax: 16384 <br />
kern.ipc.semaem: 16384 <br />
kern.ipc.semvmx: 65534 <br />
kern.ipc.semusz: 152 <br />
kern.ipc.semume: 10 <br />
kern.ipc.semopm: 100 <br />
kern.ipc.semmsl: 1024 <br />
kern.ipc.semmnu: 512 <br />
kern.ipc.semmns: 1024 <br />
kern.ipc.semmni: 1024 <br />
kern.ipc.semmap: 768 <br />
kern.ipc.shm_allow_removed: 0 <br />
kern.ipc.shm_use_phys: 1 <br />
kern.ipc.shmall: 262144 <br />
kern.ipc.shmseg: 256 <br />
kern.ipc.shmmni: 784 <br />
kern.ipc.shmmin: 1 <br />
kern.ipc.shmmax: 536870912 <br />
kern.ipc.maxsockets: 25600 <br />
<br />
gr8feen: ok...msl is max per id, mni is max ids, mns is max number of semaphores... so you probably want something like mns = mni * msl<br />
gr8feen: which one did you run out of?<br />
SDBoody: not sure how to tell- ipcs shows the sems in use add up to 1024<br />
SDBoody: there are 59 entries<br />
gr8feen: I'm assuming you tried to start postgres and it failed?<br />
SDBoody: yes<br />
gr8feen: it should have logged why, somewhere..<br />
gr8feen: if I recall, it'll tell you which one it ran out of<br />
SDBoody: > DETAIL: Failed system call was semget(1, 17, 03600).<br />
<br />
gr8feen: so it wanted an id with 17 semaphores...I'd start by making mns = 17*mni and leave mni and mnl set to what they are now and see what it does<br />
SDBoody: i think mni is plenty high<br />
SDBoody: ok, more reasonable 17408<br />
gr8feen: yeah...just change that one and see how it goes..<br />
SDBoody: and leave mni alone at 1024?<br />
gr8feen: yeah...mni id the max number of ids...but if it's trying to get something like 17 per id, your going to hit mns before you hit anything else<br />
SDBoody: right, but doesn't hurt to have it that high (assuming)<br />
<br />
gr8feen: not really... I think those get allocated out of ram that you cant page out, but it's still such a small amount that it really doesn't matter<br />
SDBoody: looks like that worked, thx!<br />
gr8feen: cool<br />
<br />
see semaphores: ipcs -a -s</pre><br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
== Making new customer VE (vm) ==<br />
<br />
This applies only to new virts >= 4.x<br />
<br />
grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.<br />
<br />
1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
also, ping the IP address.<br />
<br />
2. put CT on whichever partition has more space<br />
df -h<br />
<br />
3. vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> ) <br />
vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@johncompanies.com centos-6-x86_64 LM<br />
<br />
4. copy veid, dir, ip and password to pending customer screen. activate customer<br />
<br />
== Making new customer VE (vemakexxx) ==<br />
<br />
This applies to older virts with old templates. This should probably not be used at all anymore.<br />
<br />
<br />
1. look thru hist for ip<br />
<br />
2. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
<br />
3. put ve on whichever partition has more space<br />
vemakerh9 <veid> <ip> <hostname> <mount> <email> [gb disk]; <256|384|512> <veid><br />
vemakerh9 866 69.55.226.109 ngentu.com /vz1 ayo@ngantu.com,asd@asd.com 5; 256 866<br />
<br />
4. copy (veid), dir, and ip to pending customer screen (pass set to p455agfa)<br />
<br />
<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrateonline/migrate failures (migrate manually) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
You may also see migration failing due to quota issues.<br />
<br />
You can try to resolve by copying any quota file into the file you need:<br />
<br />
cp /var/vzquota/quota.1 /var/vzquota/quota.xxx<br />
<br />
If it complains about quota running you should then be able to stop it<br />
<br />
vzquota off xxxx<br />
<br />
If all else fails, migrate to a new VEID<br />
i.e. 1234 becomes 12341<br />
<br />
<br />
If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
'''Note:''' don't forget to fix /usr/local/sbin/backup.config for new disk location<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
=== Cant change file /etc/sysconfig/network ===<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
=== What template is user using ===<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
=== Edits needed to run java ===<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
=== Set time zone ===<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
=== Failing shm_open calls ===<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
=== Debian can't get on the network ===<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
=== CentOS can't get on the network ===<br />
<br />
When CentOS system was running, but can't get on the network, they probably updated iproute via yum.<br />
<br />
Copy an older version of the iproute rpm to their VPS.<br />
<br />
<pre>cp -p /root/iproute/iproute-2.6.32-23.el6.<i686>.rpm /vz2/private/<1527>/fs/root</pre><br />
<br />
Enter their ve, install rpm, restart network, and prevent yum from updating iproute.<br />
<br />
<pre><br />
ve <1527><br />
rpm -i --force iproute-2.6.32-23.el6.<i686>.rpm<br />
rpm -q iproute<br />
rpm -e <new iproute rpm><br />
service network restart<br />
echo "exclude=iproute" >>/etc/yum/ip-exclude<br />
echo "include=/etc/yum/ip-exclude" >>/etc/yum.conf<br />
</pre><br />
<br />
<br />
<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
----<br />
<br />
* uptime/load that vz users see isn’t what we see- it's their own load<br />
* Vz overwrites resolve.conf on every boot to what’s in ve config file. Also overwrites hostname<br />
* Quotas in vz: set Gid_limit in vz config file and restart ve. Errors leading up to this will be out of files but no inode or space issues. set to 2000 in conf, and restart vz<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
To create a new Linux VPS<br />
<br />
* get an IP address from mgmt screen<br />
* run df -h to figure out which disk has enough space<br />
* run vm with no arguments to get syntax and exact template name<br />
* use col0xxxx for hostname if they don’t give you a hostname<br />
* copy over dir, veid, ip and password to pending customer screen<br />
<br />
vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> )<br />
<br />
== disable ==<br />
To disable a customers Virtuozzo VPS server.<br />
<br />
* stop the server<br />
vzctl stop <veid><br />
* prevent restart on server reboot and user restart via Control Panel<br />
vzctl set <veid> --disabled=yes --offline_management=no --save<br />
<br />
== cancelve ==<br />
When a customer cancels all service.<br />
<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
<br />
display the contents of the vz config file for the <veid><br />
<br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
<br />
List all virtual environments running on this machine.<br />
<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
<br />
show processes running under this <veid>.<br />
<br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
<br />
Restart a <veid>.<br />
<br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
<br />
enter a <veid> as root.<br />
<br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
<br />
execute a single command in the <veid> environment.<br />
<br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem VEID <256|512|768|1024|1280|1536|2048> [burst GB]<br />
adjusts the memory resources for the VE. If this is on VZ ver >4, you will also be prompted to provide a burst ram amount. Sample usage:<br />
<br />
<pre># setmem 21671 1024 4<br />
/usr/sbin/vzctl set 21671 --ram 1024m --swap 3072m --save<br />
</pre><br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>75.80.128.147https://wiki.jcihosting.com/index.php?title=VPS_Management&diff=2227VPS Management2020-04-30T20:23:19Z<p>75.80.128.147: /* Ganeti Install a VPS */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
== Moving a server to a different account in the management system ==<br />
<br />
This is a tough one. You gotta touch a bunch of tables in the mysql database.<br />
<pre><br />
systems<br />
subs<br />
ipmap<br />
</pre><br />
<br />
basically, get the sysid and go off that:<br />
<br />
<pre><br />
update systems set cid='newcid' where sysid=xxxxx<br />
update subs set cid='newcid' where sysid=xxxxx<br />
update ipmap set cid='newcid' where sysid=xxxxx<br />
</pre><br />
<br />
You then need to rename the server disk space to change the old ID to the new one.<br />
<br />
For a jail -<br />
<pre><br />
stopjail <hostname><br />
mv 69.55.2XX.XXX-col0<oldid> 69.55.2XX.XXX-col0<newid><br />
mv 69.55.2XX.XXX-col0<oldid>-DIR 69.55.2XX.XXX-col0<newid>-DIR<br />
vi /usr/local/jail/bin/quad1 [Update the ID in the quad file]<br />
startjail <hostname><br />
</pre><br />
<br />
For a virt -<br />
You will have to <br />
* stop the virt <br />
* rename the directories <br />
* modify the config file /vzconf/<veid>.conf <br />
* rename the virt conf file <br />
* restart the new virt.<br />
<br />
that ought to do it..<br />
<br />
<br />
== Re-enable a customer account in the management system ==<br />
<br />
For example to re-instate col03438, In mysql:<br />
<br />
<pre><br />
show tables ;<br />
select * from customers where cid='col03438' ;<br />
update customers set status=1 where cid='col03438' ;<br />
select * from systems where cid='col03438' ;<br />
update systems set stop_date=NULL where cid='col03438' ;<br />
update systems set cancel_date=NULL where cid='col03438' ;<br />
update systems set status=1 where cid='col03438' ;<br />
select * from subs where cid='col03438' ;<br />
update subs set stop_date=NULL where cid='col03438' ;<br />
update subs set status=1 where cid='col03438' <br />
<br />
</pre><br />
<br />
= Ganeti VPS =<br />
<br />
This is how to create a VPS on the Lamphost ganeti cluster.<br />
I have created scripts for creating a FreeBSD VPS and one for<br />
creating a Linux VPS.<br />
<br />
== Ganeti Install a Linux VPS ==<br />
<br />
To create a linux VPS use<br />
<pre><br />
/root/new <name> <disk size> <memory> <IP Addr> [<OS>]<br />
<br />
( for example /root/new vps.example.com 30 2 69.55.238.10 centos-8 )<br />
</pre><br />
<br />
To create a FreeBSD VPS, we have to do a full install. This will create a <br />
new VPS.<br />
<br />
<pre><br />
/root/newfreebsd12 <name> <disk size> <memory> <IP Addr> [<OS>]<br />
</pre><br />
<br />
You will then have to do the install via VNC.<br />
<br />
You need to customize the install by adding a "jci" account and a customer<br />
account (col0XXXX) as well. <br />
<br />
<br> <br />
You will need to install sudo.<br />
<pre> <br />
pkg install sudo # install sudo package<br />
visudo # to add wheel account to sudoers<br />
vi /etc/group # add jci and user acct to wheel<br />
</pre><br />
<br />
<br><br />
You will also have to set up a console in /etc/ttys.<br />
<pre><br />
vi /etc/ttys<br />
</pre><br />
<br />
== Ganeti restart a VPS ==<br />
<br />
To shutdown a ganeti VPS<br />
<br />
<pre><br />
gnt-instance reboot <VPSname><br />
</pre><br />
<br />
<br />
== Ganeti shutdown a VPS ==<br />
<br />
To shutdown a ganeti VPS<br />
<br />
<pre><br />
gnt-instance shutdown <VPSname><br />
</pre><br />
<br />
To permanently delete a ganeti VPS<br />
<br />
<pre><br />
gnt-instance remove <VPSname><br />
</pre><br />
<br />
== Ganeti Re-Intsall a VPS ==<br />
<br />
Re-installing the OS will destroy all data on the VPS.<br />
Login to the Ganeti master node (currently gn1).<br />
<br />
To reinstall the OS on a VPS the VPS must be shutdown.<br />
<pre><br />
gnt-instance shutdown <VPSname><br />
</pre><br />
<br />
Do the reinstall.<br />
<pre><br />
gnt-instance reinstall <VPSname><br />
</pre><br />
<br />
Restart the VPS.<br />
<pre><br />
gnt-instance startup <VPSname><br />
</pre><br />
<br />
Now you need to do the customization for the VPS. This is<br />
basically the same as the Dedicated server customization.<br />
<br />
https://69.55.229.8/index.php/New_Signups#Dedicated_Server_Install_Checklist<br />
<br />
= FreeBSD 10.1 VPS (jail5 and jail6) =<br />
<br />
This is information on how to manage the FreeBSD 10.1 VPS servers on Jail5.<br />
<br />
<br />
== Making new customer VPS ==<br />
<pre><br />
virtmake IP[,IP] <CID> hostname ipfw# email {FSP|FB|FS|FM|FP} [size in GB]<br />
</pre><br />
<br />
Note: <br />
For now, CID must be unique in the last 2 digits of the CID.<br />
<br />
Must do a "vm start <CID>" after creating a VPS.<br />
<br />
== Starting VPS ==<br />
To start all vms on the system<br />
<pre><br />
vm start<br />
</pre><br />
<br />
To start a single VPS <br />
<pre><br />
vm start <CID><br />
</pre><br />
<br />
== List running VPS ==<br />
<pre><br />
vm status<br />
</pre><br />
<br />
== Attach to a user VPS console ==<br />
<pre><br />
vm attach <CID><br />
</pre><br />
<br />
<br />
To detach from a console, type tilde control-d (~ ctl-d).<br />
<br />
== Stop a VPS ==<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
== Restart a VPS ==<br />
<pre><br />
vm restart <CID><br />
</pre><br />
<br />
<br />
== Modify a VPS ==<br />
<br />
To modify a VPS config, stop the VPS<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
Mount the VPS data file<br />
<pre><br />
vm mount <CID><br />
<br />
cd /data/vm/private/<CID>/mount<br />
</pre><br />
<br />
Modify the user files necessary.<br />
<br />
Restart the VPS.<br />
<br />
<pre><br />
vm umount <CID><br />
<br />
vm start <CID><br />
</pre><br />
<br />
== Notes on a new FBSD 10.x VPS (jail6) ==<br />
<br />
<pre><br />
#! /bin/sh<br />
#<br />
# new <colo#> <disk><br />
<br />
<br />
memory=1024<br />
<br />
x="$1"<br />
<br />
disk="$2"<br />
if [ "y" == "y$disk" ] ; then<br />
disk=60<br />
fi<br />
<br />
disk2=`expr $disk - 4`<br />
<br />
<br />
vm image provision 04dcdf96-d843-11e6-b131-001e67fcc64f col0$x<br />
# vm configure col0$x<br />
sed -e "s/memory=256M/memory=${memory}M/" /vms/col01114/col01114.conf >/vms/col0$x/col0$x.conf<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
# vm start col0$x<br />
<br />
<br />
exit<br />
<br />
vm console col0$x<br />
<br />
#<br />
# to increase the customer disk<br />
#<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
<br />
## on the customer VPS<br />
<br />
gpart recover vtbd0<br />
gpart show vtbd0<br />
swapoff /dev/vtbd0p3<br />
gpart delete -i 3 vtbd0<br />
gpart show vtbd0<br />
sysctl kern.geom.debugflags=16<br />
gpart resize -i 2 -a 4k -s $disk2\G vtbd0<br />
<br />
growfs /dev/vtbd0p2<br />
<br />
gpart show vtbd0<br />
gpart add -t freebsd-swap -a 4k vtbd0<br />
gpart show vtbd0<br />
swapon /dev/vtbd0p3<br />
<br />
df -h<br />
reboot<br />
<br />
vi /etc/rc.conf<br />
hostname<br />
IP Addr<br />
passwd root<br />
passwd user<br />
clear history<br />
cp /dev/null /var/log/utx.log<br />
cp /dev/null ~root/.history<br />
history -c<br />
<br />
</pre><br />
<br />
= FreeBSD Jails =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Making new customer jail 4.x ==<br />
<br />
1. run [[#js|js]] to figure out which partition, IP to put it on, find an unused vn, and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vnX hostname shorthost quad/safe# ipfw# email [gb disk, default 4]</tt><br />
<br />
jail14# /tmp/jailmake 69.55.226.152 /mnt/data2/69.55.226.152-col00182 vn23 negev.cerastes.org negev 4 182 cerastes@cerastes.org<br />
<br />
<br />
== Making new customer jail 6.x ==<br />
<br />
1. run [[#js|js]] to figure out which gvinum vols are free (and which mnt (disk) each gvinum goes with/to), IP to put it on and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vN[,vN] hostname shorthost quad/safe# ipfw# email</tt><br />
<br />
jail19# jailmake 69.55.236.17 /mnt/data1/69.55.236.17-col01574 v10 uncle-leo.commadev.com uncle-leo 1 1574 lee@commadev.com,lee@gmail.com<br />
<br />
<br />
== Changing an ip for freebsd VPS ==<br />
<br />
*stopjail (hostname)<br />
*on systems using vnfile or mdfile: rename vnfile/mdfile, using new ip<br />
*rename dir using new ip<br />
*Edit quad (make sure to update all lines: directory, vn/mdfile)<br />
*buildsafe (not necessary on systems having only quad1)<br />
*if you're adding an ip not already on the host machine: <tt>ipadd x.x.x.x</tt><br />
*startjail (hostname)<br />
if backups: <tt>mvbackups</tt> ...<br />
*Edit dir/ip in mgmt<br />
*if any rules, update any firewall rules to use new ip (careful not to make sure there arent multiple rules per ipfw#- search by colo#)<br />
*(if customer asks and has domains) update their domains on ns1c<br />
*optional: update ipfw counters on the host server<br />
<br />
<br />
== Rename a gconcat vol ==<br />
<br />
*stop jail, unmount everything<br />
*gconcat stop vol<br />
*gconcat clear /dev/gvinum/vN<br />
*gconcat clear /dev/gvinum/vN+1<br />
*gconcat label -v newname /dev/gvinum/vN /dev/gvinum/vN+1<br />
*bsdlabel /dev/concat/newname<br />
(make sure a partition is not fstype unused, if so change to: 4.2BSD)<br />
<br />
<br />
== Remaking a VPS (on same jail) ==<br />
<br />
# take him down (stopjail)<br />
# make note of host, vn/gvin/md, ip<br />
# remove from quad/safe<br />
# move vnfile (rename)<br />
# remove dir (jailmake will make a new one)<br />
# remove ipfw counts<br />
# jailmake<br />
# update backup script<br />
# update db with new dir/ip (use the “move” link, mark old as stopped today)<br />
# update firewall if changed ip<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
<br />
== Adding disk to a 4.x jail ==<br />
<br />
NOTE: if you unmount before vnconfig, re-vnconfig then unmount then vnunconfig<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them.<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to dump to a different partition and create the new vn on the current partition. If no space to make make dump, then move old vn’s to backup or worse, dump to backup.<br />
<br />
<br />
2. <tt>g <customerID></tt> to get the info<br />
<br />
<br />
3. make vnfile, newfs and mount. USE CAUTION not to overwrite existing vn. <br><br />
Ex: <tt>vnconfig –T -S 9g -s labels -c /dev/vn22 /mnt/data2/65.214.160.117-col00496<br><br />
disklabel -r -w vn30 auto<br><br />
newfs /dev/vn30c<br><br />
mkdir /mnt/data2/65.214.160.117-col00496-DIR<br><br />
mount /dev/vn30c /mnt/data2/69.55.237.26-col00241-DIR<br><br />
cd /mnt/data2/65.214.160.117-col00496-DIR</tt><br />
<br />
<br />
4. <tt>jailkill <hostname></tt><br />
<br />
<br />
5. <tt>dump -0a -f - /dev/vn1 | restore -r -f -</tt> <br />
<br />
<br />
6. <tt>rm restoresymtable</tt><br />
<br />
<br />
7. unmount and unvnconfig old system:<br><br />
<tt>umount /dev/vnNNc <br><br />
vnconfig –u /dev/vnNN</tt><br />
<br />
<br />
8. move vnfile. Ex: <tt>mv /mnt/data1/69.55.237.26-col00241 /mnt/data1/old-col00241-vnfile-20110312-noarchive</tt><br />
<br />
<br />
9. remove old dir: <tt>rmdir /mnt/data1/69.55.237.26-col00241-DIR</tt><br />
<br />
<br />
10. edit quad/safe to point to new location, run <tt>[[#buildsafe|buildsafe]]</tt><br />
<br />
<br />
11. start jail: <tt>[[#startjail|startjail]] <hostname></tt><br />
<br />
<br />
12. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
13. mv backups if necessary<br />
<br />
WARNING: if you restore to unmounted vn then you’re actually restoring to /mnt/data1<br />
you can tar and do above to recover<br />
<br />
NOTE: you can’t move a customer to a system that has a base sys that is diff from the one they came from (BSD 4.8 to 4.5 = won’t work)<br />
<br />
NOTE: we don’t mount procfs's to anyone’s system by default.<br />
<br />
NOTE: mount_nulls: <br />
also goofy mount_nulls seen in df -k on jail2<br><br />
also needs to be done in safe and quad<br><br />
if he wants another, use safe/quad as example and add it to file. also exec command at prompt to add mount (no reboot, remount or rejail necessary)<br />
<br />
== Moving customer to another jail machine ==<br />
<br />
Systems should only be moved between similarly-versioned jails. If you cannot get a match, you should go to freebsd.org and see what the differences were between the version you're moving from and the new version. Many of the updates will be to drivers and kernels - these don't affect the customer. Perhaps they updated traceroute or a library...you should copy the new/changed files into the VPS/jail on the new host. An alternate method for moving to a different version would be to buildworld to bring their full distribution up to date. Optional steps for this kind of upgrade are included below.<br />
<br />
1. <tt>g <customerID></tt><br />
<br />
<br />
2. <tt>[[#jailkill|jailkill]] <hostname></tt><br />
<br />
<br />
3. create new device on target system<br><br />
4.x:<br><br />
<pre>vnconfig –T –S 4g –s labels –c /dev/vn1 /mnt/data1/69.55.22x.x-col00XXX<br />
disklabel –r –w vn1 auto<br />
newfs /dev/vn1c</pre><br />
<br />
6.x:<br><br />
<pre>bsdlabel -r -w /dev/gvinum/v1<br />
newfs /dev/gvinum/v1a<br />
or <br />
gconcat label v1-v3 /dev/gvinum/v1 /dev/gvinum/v2 /dev/gvinum/v3<br />
bsdlabel -r -w /dev/concat/v1-v3<br />
newfs /dev/concat/v1-v3a</pre><br />
<br />
7.x+:<br><br />
Run jailmakeempty then skip step 4 & 7 below<br />
<br />
<br />
4. make and mount the dir on new system. <br><br />
Ex: <pre>mkdir /mnt/data2/69.55.230.3-col00123-DIR<br />
chmod 755 /mnt/data2/69.55.230.3-col00123-DIR<br />
mount [device] /mnt/data2/69.55.230.3-col00123-DIR</pre><br />
<br />
<br />
5. [[#stopjail|stopjail]] <hostname> 1<br />
<br />
<br />
6. dump fs to new system. ex: <tt>[[#dumpremoterestore|dumpremoterestore]] /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR</tt> (make sure you can ssh as root on remote machine) <br />
<br />
<br />
6a. OPTIONAL BUILDWORLD:<br><br />
cd /usr/src<br />
make world DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
cd etc<br />
make distribution DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
(you may have to rm an openssh file and re-make dist)<br />
<br />
rm -rf /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/periodic/daily/400.status-disks<br />
vi /etc/periodic/security/100.chksetuid<br />
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt><br><br />
with: <tt>MP='/' (use single quotes)</tt><br />
<br />
Ask user if they want ports overwritten with current, if yes:<br />
cp -r /usr/ports /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/usr<br />
<br />
on source: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d .<br />
(copy in all info)<br />
<br />
on target: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d . <br />
(paste all info)<br />
<br />
on source: <br />
cat /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group<br />
(copy in all info)<br />
<br />
on target:<br />
cat > /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group <br />
(paste all info)<br />
<br />
<br />
7. edit quad on source system copy over entries for the jail to the target system <br />
take care that the vn/gvinum/gconcat/md devices on the target system are’t in use and that the /mnt/dataN path doesn’t need changing<br />
<br />
<br />
8. run [[#buildsafe|buildsafe]] (if this is <=6.x) on target system (copies changes made to quad into safe file)<br />
<br />
<br />
9. remove ip from source system. Ex: <tt>ipdel 69.55.230.3</tt><br />
<br />
<br />
10. add ip to target system. Ex: <tt>ipadd 69.55.230.3</tt><br />
<br />
<br />
11. start new system, with [[#startjail|startjail]] or manually, pasting entries found from running <tt>g <customerID></tt> on the new system<br />
<br />
<br />
12. run <tt>[[#canceljail|canceljail]] col0xxxx</tt> on source system. This should prompt you to remove backups if any existed. Do not let it do this, rather:<br />
<br />
<br />
13. if backups existed move them to the new host via the <tt>[[#mvbackups_.28freebsd.29|mvbackups]] script</tt><br />
<br />
<br />
14. edit quad on source system, edit comment to reflect a move rather than a cancel, ex: <tt># moved to jail2 col00241</tt><br />
<br />
<br />
15. edit mgmt to reflect new host and dir for new system<br />
<br />
<br />
16. optional add ipfw rules on new system. Ex:<br />
<tt>ipfw add 01231 count ip from 69.55.230.2 to any<br />
ipfw add 01232 count ip from any to 69.55.230.2</tt><br />
<br />
== Increasing inodes for a VPS ==<br />
<br />
when doing the newfs: <br />
newfs -i 4096 ...<br />
<br />
<br />
== Jail services unable to contact services within same jail ==<br />
<br />
This is due to a messed up routing table (only seen it on jail2 before). <br />
<br />
route delete 69.55.228.65/32 route add -net 69.55.228.65/32 -iface fxp0 -nostatic -cloning route add 69.55.228.65 -iface lo0<br />
<br />
<br />
== /dev/null permission resets == <br />
<br />
Applies to 4.x jails only<br />
<br />
For some reason that I do not understand at all, the /dev/null node in customer jails, after the system is restarted, often reverts permissions to 0600 ... which is bad, because non-root processes cannot redirect to /dev/null anymore. In fact, a fair number of server daemons will complain and/or fail if this is the case.<br />
<br />
So, after a restart, after all systems have restarted, run postboot, which does something like:<br />
<br />
for f in `df -k | grep /dev/vn | awk '{print $6}'` ; do chmod 0666 $f/dev/null ; done<br />
<br />
Later, after the system is up, if anyone ever complains about their /dev/null permissions for any reason ... really if anyone mentions<br />
/dev/null in any way, tell them to run:<br />
<br />
chmod 0666 /dev/null<br />
<br />
We have ultimately fixed this by adding chmods to the quad/safe's<br />
<br />
<br />
== mknod in a jail == <br />
<br />
For a customer who wants to run a chroot name server (named) in a jail.<br />
<br />
<pre><br />
cd /mnt/data1/<user directory>/var/named<br />
mknod -c 1 1 null<br />
mknod -c 2 3 random<br />
</pre><br />
<br />
== Postfix Problems ==<br />
<br />
This hasn't happened in a long time, probably no longer valid with modern OS's<br />
<br />
Postfix is an alternate MTA - people replace sendmail with it in much the same way that many people also replace sendmail with qmail. The problem is that if you install postfix inside of a jail, by default it will not work properly - the user will see error messages like this:<br />
<br />
incoming mail generates errors like this:<br />
<br />
mail_queue_enter: create file incoming/298266.55902: File too large<br />
<br />
and further, sending mail generates errors like this:<br />
<br />
postdrop: warning: mail_queue_enter: create file maildrop/627930.56676: File too large<br />
<br />
This is very easy to solve. In fact, the FreeBSD welcome email that is sent out by `jailmake` now contains this block of text:<br />
<br />
- if you plan on installing postfix, email us first and ask for a necessary patch.<br />
<br />
So, anytime you see a support email that says anything about postfix not working, paste that line from the welcome email into your response to them, and right away, forward the postfix patch to them in an email.<br />
<br />
Here are the instructions:<br />
<br />
Simply copy the patch into /usr/ports/mail/postfix/files, rename the file to ‘patch-file_limit.c’ and then cd to /usr/ports/mail/postfix and run `make install`.<br />
<br />
The postfix patch is the very first email in the support email box, and we never delete or save it because we always want it there to forward to people. The instructions in the email are very clear, and it solves their problem 100% of the time.<br />
<br />
<br />
== Problems with `find` ==<br />
<br />
There was a bug in the FreeBSD vn-filesystem code. The bug causes the system to crash or hang when someone runs the `find` command inside their vn-backed filesystem (and even though the bug may have been fixed, due to the i/o overhead, we still don't like find).<br />
<br />
Now, this does not always happen - however, any jail machine with more than 20 systems on it will crash every night if every system on it runs the daily periodic script out of their crontab, and updates their locate database with the find command.<br />
<br />
(the locate database is a small, FreeBSD specific database that is populated nightly from a cron job, and is then used to provide fast answers to the `locate` command)<br />
<br />
In normal systems (32-40 jails on them), all that has to be done is make sure nobody runs the daily periodic. This means that the file /etc/crontab, instead of looking like this:<br />
<br />
<pre># do daily/weekly/monthly maintenance<br />
1 3 * * * root periodic daily<br />
15 4 * * 6 root periodic weekly<br />
30 5 1 * * root periodic monthly</pre><br />
<br />
Needs to look like this:<br />
<br />
<pre># DO NOT UNCOMMENT THESE - contact support@johncompanies.com for details<br />
#1 3 * * * root periodic daily<br />
#15 4 * * 6 root periodic weekly<br />
#30 5 1 * * root periodic monthly<br />
## DO NOT UNCOMMENT THESE ^^^^^^^^</pre><br />
<br />
and the problem will generally not occur at all. New filesystem images are always altered in this way, so all systems are set like this.<br />
<br />
However, if you have enough systems running, even casual uses of find, outside of the periodic scripts, can cause the system to crash. We are seeing this on jail13, which crashes about once per week.<br />
<br />
There is not yet any resolution to this problem.<br />
<br />
So, if you have a freebsd system crash, see if anyone has uncommented their daily periodic line - you can see them for everyone by running:<br />
<br />
/bin/sh<br />
<br />
and then running:<br />
<br />
<pre>for f in `df -k | grep vn | awk '{print $6}'` ; do echo `cat $f/etc/crontab | grep "periodic daily"` $f ; done | more</pre><br />
<br />
(all one line, one command)<br />
<br />
And then edit those /etc/crontabs and comment them out again.<br />
<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
* if you are getting disk full messages for a BSD customer, it's fairly safe to clear out their /usr/ports/distfiles dir<br />
* 4.x: ps and top an only be run by root in these jails. Done on purpose:<br />
As for `ps` and `top` - non-root users can run them, just not successfully because we have locked the permissions on /dev/mem and /dev/kmem to be root-readable only. That is why non-root users cannot successfully run `ps` and `top`.<br />
* user quotas do not work on freebsd jails - you cannot set up quotas at all, and that's that.<br />
* You cannot inject a process into a 4.x jail, only HUP running processes. 6.x and onward you can with jexec<br />
* jails see base machine’s uptime/load when running top/w<br />
* if someone is unable to get in- cant ping, etc- see if they were blocked by castle (DoS), see if their ip is on the system (post reboot it was lost cause wasn’t in the rc.conf). preboot should catch that.<br />
* in FreeBSD you cant su to root unless you belong to wheel group – so if you remove your acct and setup a new one, we have to add it to wheel group (add to /etc/group)<br />
* Dmesg from underlying sys is seen in customers dmesg on jail<br />
* Popper process going crazy @ 40% for 10min = someone who leaves mail on server<br />
* Don’t force umounts on 4.x jails – it crashes the machine, generally ok on newer machines (running md)<br />
* Good book for admin http://search.barnesandnoble.com/booksearch/isbnInquiry.asp?userid=t824VyRAYz&isbn=0596005164&itm=2<br />
* Self-sign ssl cert http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#selfcert<br />
* conversation with Glenn about semaphores and pgsql<br />
<pre>SDBoody: hey, these are valid amounts/figures/increments right:<br />
kern.ipc.semmni=1280<br />
kern.ipc.semmns=1280<br />
<br />
gr8feen: probably... I always forget exactly what those are, so I usually have to look them up<br />
SDBoody: semaphores<br />
SDBoody: i took the current 1024 and added 256 to them<br />
SDBoody: need more for pgsql<br />
gr8feen: I meant the mni and mns parts...<br />
gr8feen: some of those are not ones you just want to add to... hang on a sec and I'll look them up..<br />
gr8feen: what's semmsl set to?<br />
SDBoody: kern.ipc.semmsl: 1024<br />
<br />
SDBoody: kern.ipc.msgseg: 2048 <br />
kern.ipc.msgssz: 8 <br />
kern.ipc.msgtql: 40 <br />
kern.ipc.msgmnb: 2048 <br />
kern.ipc.msgmni: 40 <br />
kern.ipc.msgmax: 16384 <br />
kern.ipc.semaem: 16384 <br />
kern.ipc.semvmx: 65534 <br />
kern.ipc.semusz: 152 <br />
kern.ipc.semume: 10 <br />
kern.ipc.semopm: 100 <br />
kern.ipc.semmsl: 1024 <br />
kern.ipc.semmnu: 512 <br />
kern.ipc.semmns: 1024 <br />
kern.ipc.semmni: 1024 <br />
kern.ipc.semmap: 768 <br />
kern.ipc.shm_allow_removed: 0 <br />
kern.ipc.shm_use_phys: 1 <br />
kern.ipc.shmall: 262144 <br />
kern.ipc.shmseg: 256 <br />
kern.ipc.shmmni: 784 <br />
kern.ipc.shmmin: 1 <br />
kern.ipc.shmmax: 536870912 <br />
kern.ipc.maxsockets: 25600 <br />
<br />
gr8feen: ok...msl is max per id, mni is max ids, mns is max number of semaphores... so you probably want something like mns = mni * msl<br />
gr8feen: which one did you run out of?<br />
SDBoody: not sure how to tell- ipcs shows the sems in use add up to 1024<br />
SDBoody: there are 59 entries<br />
gr8feen: I'm assuming you tried to start postgres and it failed?<br />
SDBoody: yes<br />
gr8feen: it should have logged why, somewhere..<br />
gr8feen: if I recall, it'll tell you which one it ran out of<br />
SDBoody: > DETAIL: Failed system call was semget(1, 17, 03600).<br />
<br />
gr8feen: so it wanted an id with 17 semaphores...I'd start by making mns = 17*mni and leave mni and mnl set to what they are now and see what it does<br />
SDBoody: i think mni is plenty high<br />
SDBoody: ok, more reasonable 17408<br />
gr8feen: yeah...just change that one and see how it goes..<br />
SDBoody: and leave mni alone at 1024?<br />
gr8feen: yeah...mni id the max number of ids...but if it's trying to get something like 17 per id, your going to hit mns before you hit anything else<br />
SDBoody: right, but doesn't hurt to have it that high (assuming)<br />
<br />
gr8feen: not really... I think those get allocated out of ram that you cant page out, but it's still such a small amount that it really doesn't matter<br />
SDBoody: looks like that worked, thx!<br />
gr8feen: cool<br />
<br />
see semaphores: ipcs -a -s</pre><br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
== Making new customer VE (vm) ==<br />
<br />
This applies only to new virts >= 4.x<br />
<br />
grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.<br />
<br />
1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
also, ping the IP address.<br />
<br />
2. put CT on whichever partition has more space<br />
df -h<br />
<br />
3. vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> ) <br />
vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@johncompanies.com centos-6-x86_64 LM<br />
<br />
4. copy veid, dir, ip and password to pending customer screen. activate customer<br />
<br />
== Making new customer VE (vemakexxx) ==<br />
<br />
This applies to older virts with old templates. This should probably not be used at all anymore.<br />
<br />
<br />
1. look thru hist for ip<br />
<br />
2. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
<br />
3. put ve on whichever partition has more space<br />
vemakerh9 <veid> <ip> <hostname> <mount> <email> [gb disk]; <256|384|512> <veid><br />
vemakerh9 866 69.55.226.109 ngentu.com /vz1 ayo@ngantu.com,asd@asd.com 5; 256 866<br />
<br />
4. copy (veid), dir, and ip to pending customer screen (pass set to p455agfa)<br />
<br />
<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrateonline/migrate failures (migrate manually) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
You may also see migration failing due to quota issues.<br />
<br />
You can try to resolve by copying any quota file into the file you need:<br />
<br />
cp /var/vzquota/quota.1 /var/vzquota/quota.xxx<br />
<br />
If it complains about quota running you should then be able to stop it<br />
<br />
vzquota off xxxx<br />
<br />
If all else fails, migrate to a new VEID<br />
i.e. 1234 becomes 12341<br />
<br />
<br />
If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
'''Note:''' don't forget to fix /usr/local/sbin/backup.config for new disk location<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
=== Cant change file /etc/sysconfig/network ===<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
=== What template is user using ===<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
=== Edits needed to run java ===<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
=== Set time zone ===<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
=== Failing shm_open calls ===<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
=== Debian can't get on the network ===<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
=== CentOS can't get on the network ===<br />
<br />
When CentOS system was running, but can't get on the network, they probably updated iproute via yum.<br />
<br />
Copy an older version of the iproute rpm to their VPS.<br />
<br />
<pre>cp -p /root/iproute/iproute-2.6.32-23.el6.<i686>.rpm /vz2/private/<1527>/fs/root</pre><br />
<br />
Enter their ve, install rpm, restart network, and prevent yum from updating iproute.<br />
<br />
<pre><br />
ve <1527><br />
rpm -i --force iproute-2.6.32-23.el6.<i686>.rpm<br />
rpm -q iproute<br />
rpm -e <new iproute rpm><br />
service network restart<br />
echo "exclude=iproute" >>/etc/yum/ip-exclude<br />
echo "include=/etc/yum/ip-exclude" >>/etc/yum.conf<br />
</pre><br />
<br />
<br />
<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
----<br />
<br />
* uptime/load that vz users see isn’t what we see- it's their own load<br />
* Vz overwrites resolve.conf on every boot to what’s in ve config file. Also overwrites hostname<br />
* Quotas in vz: set Gid_limit in vz config file and restart ve. Errors leading up to this will be out of files but no inode or space issues. set to 2000 in conf, and restart vz<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
To create a new Linux VPS<br />
<br />
* get an IP address from mgmt screen<br />
* run df -h to figure out which disk has enough space<br />
* run vm with no arguments to get syntax and exact template name<br />
* use col0xxxx for hostname if they don’t give you a hostname<br />
* copy over dir, veid, ip and password to pending customer screen<br />
<br />
vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> )<br />
<br />
== disable ==<br />
To disable a customers Virtuozzo VPS server.<br />
<br />
* stop the server<br />
vzctl stop <veid><br />
* prevent restart on server reboot and user restart via Control Panel<br />
vzctl set <veid> --disabled=yes --offline_management=no --save<br />
<br />
== cancelve ==<br />
When a customer cancels all service.<br />
<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
<br />
display the contents of the vz config file for the <veid><br />
<br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
<br />
List all virtual environments running on this machine.<br />
<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
<br />
show processes running under this <veid>.<br />
<br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
<br />
Restart a <veid>.<br />
<br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
<br />
enter a <veid> as root.<br />
<br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
<br />
execute a single command in the <veid> environment.<br />
<br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem VEID <256|512|768|1024|1280|1536|2048> [burst GB]<br />
adjusts the memory resources for the VE. If this is on VZ ver >4, you will also be prompted to provide a burst ram amount. Sample usage:<br />
<br />
<pre># setmem 21671 1024 4<br />
/usr/sbin/vzctl set 21671 --ram 1024m --swap 3072m --save<br />
</pre><br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>75.80.128.147https://wiki.jcihosting.com/index.php?title=VPS_Management&diff=2226VPS Management2020-04-30T20:21:50Z<p>75.80.128.147: /* Ganeti shutdown a VPS */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
== Moving a server to a different account in the management system ==<br />
<br />
This is a tough one. You gotta touch a bunch of tables in the mysql database.<br />
<pre><br />
systems<br />
subs<br />
ipmap<br />
</pre><br />
<br />
basically, get the sysid and go off that:<br />
<br />
<pre><br />
update systems set cid='newcid' where sysid=xxxxx<br />
update subs set cid='newcid' where sysid=xxxxx<br />
update ipmap set cid='newcid' where sysid=xxxxx<br />
</pre><br />
<br />
You then need to rename the server disk space to change the old ID to the new one.<br />
<br />
For a jail -<br />
<pre><br />
stopjail <hostname><br />
mv 69.55.2XX.XXX-col0<oldid> 69.55.2XX.XXX-col0<newid><br />
mv 69.55.2XX.XXX-col0<oldid>-DIR 69.55.2XX.XXX-col0<newid>-DIR<br />
vi /usr/local/jail/bin/quad1 [Update the ID in the quad file]<br />
startjail <hostname><br />
</pre><br />
<br />
For a virt -<br />
You will have to <br />
* stop the virt <br />
* rename the directories <br />
* modify the config file /vzconf/<veid>.conf <br />
* rename the virt conf file <br />
* restart the new virt.<br />
<br />
that ought to do it..<br />
<br />
<br />
== Re-enable a customer account in the management system ==<br />
<br />
For example to re-instate col03438, In mysql:<br />
<br />
<pre><br />
show tables ;<br />
select * from customers where cid='col03438' ;<br />
update customers set status=1 where cid='col03438' ;<br />
select * from systems where cid='col03438' ;<br />
update systems set stop_date=NULL where cid='col03438' ;<br />
update systems set cancel_date=NULL where cid='col03438' ;<br />
update systems set status=1 where cid='col03438' ;<br />
select * from subs where cid='col03438' ;<br />
update subs set stop_date=NULL where cid='col03438' ;<br />
update subs set status=1 where cid='col03438' <br />
<br />
</pre><br />
<br />
= Ganeti VPS =<br />
<br />
This is how to create a VPS on the Lamphost ganeti cluster.<br />
I have created scripts for creating a FreeBSD VPS and one for<br />
creating a Linux VPS.<br />
<br />
== Ganeti Install a VPS ==<br />
<br />
To create a linux VPS use<br />
<pre><br />
/root/new <name> <disk size> <memory> <IP Addr> [<OS>]<br />
<br />
( for example /root/new vps.example.com 30 2 69.55.238.10 centos-8 )<br />
</pre><br />
<br />
To create a FreeBSD VPS, we have to do a full install. This will create a <br />
new VPS.<br />
<br />
<pre><br />
/root/newfreebsd12 <name> <disk size> <memory> <IP Addr> [<OS>]<br />
</pre><br />
<br />
You will then have to do the install via VNC.<br />
<br />
You need to customize the install by adding a "jci" account and a customer<br />
account (col0XXXX) as well. <br />
<br />
<br> <br />
You will need to install sudo.<br />
<pre> <br />
pkg install sudo # install sudo package<br />
visudo # to add wheel account to sudoers<br />
vi /etc/group # add jci and user acct to wheel<br />
</pre><br />
<br />
<br><br />
You will also have to set up a console in /etc/ttys.<br />
<pre><br />
vi /etc/ttys<br />
</pre><br />
<br />
== Ganeti restart a VPS ==<br />
<br />
To shutdown a ganeti VPS<br />
<br />
<pre><br />
gnt-instance reboot <VPSname><br />
</pre><br />
<br />
<br />
== Ganeti shutdown a VPS ==<br />
<br />
To shutdown a ganeti VPS<br />
<br />
<pre><br />
gnt-instance shutdown <VPSname><br />
</pre><br />
<br />
To permanently delete a ganeti VPS<br />
<br />
<pre><br />
gnt-instance remove <VPSname><br />
</pre><br />
<br />
== Ganeti Re-Intsall a VPS ==<br />
<br />
Re-installing the OS will destroy all data on the VPS.<br />
Login to the Ganeti master node (currently gn1).<br />
<br />
To reinstall the OS on a VPS the VPS must be shutdown.<br />
<pre><br />
gnt-instance shutdown <VPSname><br />
</pre><br />
<br />
Do the reinstall.<br />
<pre><br />
gnt-instance reinstall <VPSname><br />
</pre><br />
<br />
Restart the VPS.<br />
<pre><br />
gnt-instance startup <VPSname><br />
</pre><br />
<br />
Now you need to do the customization for the VPS. This is<br />
basically the same as the Dedicated server customization.<br />
<br />
https://69.55.229.8/index.php/New_Signups#Dedicated_Server_Install_Checklist<br />
<br />
= FreeBSD 10.1 VPS (jail5 and jail6) =<br />
<br />
This is information on how to manage the FreeBSD 10.1 VPS servers on Jail5.<br />
<br />
<br />
== Making new customer VPS ==<br />
<pre><br />
virtmake IP[,IP] <CID> hostname ipfw# email {FSP|FB|FS|FM|FP} [size in GB]<br />
</pre><br />
<br />
Note: <br />
For now, CID must be unique in the last 2 digits of the CID.<br />
<br />
Must do a "vm start <CID>" after creating a VPS.<br />
<br />
== Starting VPS ==<br />
To start all vms on the system<br />
<pre><br />
vm start<br />
</pre><br />
<br />
To start a single VPS <br />
<pre><br />
vm start <CID><br />
</pre><br />
<br />
== List running VPS ==<br />
<pre><br />
vm status<br />
</pre><br />
<br />
== Attach to a user VPS console ==<br />
<pre><br />
vm attach <CID><br />
</pre><br />
<br />
<br />
To detach from a console, type tilde control-d (~ ctl-d).<br />
<br />
== Stop a VPS ==<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
== Restart a VPS ==<br />
<pre><br />
vm restart <CID><br />
</pre><br />
<br />
<br />
== Modify a VPS ==<br />
<br />
To modify a VPS config, stop the VPS<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
Mount the VPS data file<br />
<pre><br />
vm mount <CID><br />
<br />
cd /data/vm/private/<CID>/mount<br />
</pre><br />
<br />
Modify the user files necessary.<br />
<br />
Restart the VPS.<br />
<br />
<pre><br />
vm umount <CID><br />
<br />
vm start <CID><br />
</pre><br />
<br />
== Notes on a new FBSD 10.x VPS (jail6) ==<br />
<br />
<pre><br />
#! /bin/sh<br />
#<br />
# new <colo#> <disk><br />
<br />
<br />
memory=1024<br />
<br />
x="$1"<br />
<br />
disk="$2"<br />
if [ "y" == "y$disk" ] ; then<br />
disk=60<br />
fi<br />
<br />
disk2=`expr $disk - 4`<br />
<br />
<br />
vm image provision 04dcdf96-d843-11e6-b131-001e67fcc64f col0$x<br />
# vm configure col0$x<br />
sed -e "s/memory=256M/memory=${memory}M/" /vms/col01114/col01114.conf >/vms/col0$x/col0$x.conf<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
# vm start col0$x<br />
<br />
<br />
exit<br />
<br />
vm console col0$x<br />
<br />
#<br />
# to increase the customer disk<br />
#<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
<br />
## on the customer VPS<br />
<br />
gpart recover vtbd0<br />
gpart show vtbd0<br />
swapoff /dev/vtbd0p3<br />
gpart delete -i 3 vtbd0<br />
gpart show vtbd0<br />
sysctl kern.geom.debugflags=16<br />
gpart resize -i 2 -a 4k -s $disk2\G vtbd0<br />
<br />
growfs /dev/vtbd0p2<br />
<br />
gpart show vtbd0<br />
gpart add -t freebsd-swap -a 4k vtbd0<br />
gpart show vtbd0<br />
swapon /dev/vtbd0p3<br />
<br />
df -h<br />
reboot<br />
<br />
vi /etc/rc.conf<br />
hostname<br />
IP Addr<br />
passwd root<br />
passwd user<br />
clear history<br />
cp /dev/null /var/log/utx.log<br />
cp /dev/null ~root/.history<br />
history -c<br />
<br />
</pre><br />
<br />
= FreeBSD Jails =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Making new customer jail 4.x ==<br />
<br />
1. run [[#js|js]] to figure out which partition, IP to put it on, find an unused vn, and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vnX hostname shorthost quad/safe# ipfw# email [gb disk, default 4]</tt><br />
<br />
jail14# /tmp/jailmake 69.55.226.152 /mnt/data2/69.55.226.152-col00182 vn23 negev.cerastes.org negev 4 182 cerastes@cerastes.org<br />
<br />
<br />
== Making new customer jail 6.x ==<br />
<br />
1. run [[#js|js]] to figure out which gvinum vols are free (and which mnt (disk) each gvinum goes with/to), IP to put it on and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vN[,vN] hostname shorthost quad/safe# ipfw# email</tt><br />
<br />
jail19# jailmake 69.55.236.17 /mnt/data1/69.55.236.17-col01574 v10 uncle-leo.commadev.com uncle-leo 1 1574 lee@commadev.com,lee@gmail.com<br />
<br />
<br />
== Changing an ip for freebsd VPS ==<br />
<br />
*stopjail (hostname)<br />
*on systems using vnfile or mdfile: rename vnfile/mdfile, using new ip<br />
*rename dir using new ip<br />
*Edit quad (make sure to update all lines: directory, vn/mdfile)<br />
*buildsafe (not necessary on systems having only quad1)<br />
*if you're adding an ip not already on the host machine: <tt>ipadd x.x.x.x</tt><br />
*startjail (hostname)<br />
if backups: <tt>mvbackups</tt> ...<br />
*Edit dir/ip in mgmt<br />
*if any rules, update any firewall rules to use new ip (careful not to make sure there arent multiple rules per ipfw#- search by colo#)<br />
*(if customer asks and has domains) update their domains on ns1c<br />
*optional: update ipfw counters on the host server<br />
<br />
<br />
== Rename a gconcat vol ==<br />
<br />
*stop jail, unmount everything<br />
*gconcat stop vol<br />
*gconcat clear /dev/gvinum/vN<br />
*gconcat clear /dev/gvinum/vN+1<br />
*gconcat label -v newname /dev/gvinum/vN /dev/gvinum/vN+1<br />
*bsdlabel /dev/concat/newname<br />
(make sure a partition is not fstype unused, if so change to: 4.2BSD)<br />
<br />
<br />
== Remaking a VPS (on same jail) ==<br />
<br />
# take him down (stopjail)<br />
# make note of host, vn/gvin/md, ip<br />
# remove from quad/safe<br />
# move vnfile (rename)<br />
# remove dir (jailmake will make a new one)<br />
# remove ipfw counts<br />
# jailmake<br />
# update backup script<br />
# update db with new dir/ip (use the “move” link, mark old as stopped today)<br />
# update firewall if changed ip<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
<br />
== Adding disk to a 4.x jail ==<br />
<br />
NOTE: if you unmount before vnconfig, re-vnconfig then unmount then vnunconfig<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them.<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to dump to a different partition and create the new vn on the current partition. If no space to make make dump, then move old vn’s to backup or worse, dump to backup.<br />
<br />
<br />
2. <tt>g <customerID></tt> to get the info<br />
<br />
<br />
3. make vnfile, newfs and mount. USE CAUTION not to overwrite existing vn. <br><br />
Ex: <tt>vnconfig –T -S 9g -s labels -c /dev/vn22 /mnt/data2/65.214.160.117-col00496<br><br />
disklabel -r -w vn30 auto<br><br />
newfs /dev/vn30c<br><br />
mkdir /mnt/data2/65.214.160.117-col00496-DIR<br><br />
mount /dev/vn30c /mnt/data2/69.55.237.26-col00241-DIR<br><br />
cd /mnt/data2/65.214.160.117-col00496-DIR</tt><br />
<br />
<br />
4. <tt>jailkill <hostname></tt><br />
<br />
<br />
5. <tt>dump -0a -f - /dev/vn1 | restore -r -f -</tt> <br />
<br />
<br />
6. <tt>rm restoresymtable</tt><br />
<br />
<br />
7. unmount and unvnconfig old system:<br><br />
<tt>umount /dev/vnNNc <br><br />
vnconfig –u /dev/vnNN</tt><br />
<br />
<br />
8. move vnfile. Ex: <tt>mv /mnt/data1/69.55.237.26-col00241 /mnt/data1/old-col00241-vnfile-20110312-noarchive</tt><br />
<br />
<br />
9. remove old dir: <tt>rmdir /mnt/data1/69.55.237.26-col00241-DIR</tt><br />
<br />
<br />
10. edit quad/safe to point to new location, run <tt>[[#buildsafe|buildsafe]]</tt><br />
<br />
<br />
11. start jail: <tt>[[#startjail|startjail]] <hostname></tt><br />
<br />
<br />
12. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
13. mv backups if necessary<br />
<br />
WARNING: if you restore to unmounted vn then you’re actually restoring to /mnt/data1<br />
you can tar and do above to recover<br />
<br />
NOTE: you can’t move a customer to a system that has a base sys that is diff from the one they came from (BSD 4.8 to 4.5 = won’t work)<br />
<br />
NOTE: we don’t mount procfs's to anyone’s system by default.<br />
<br />
NOTE: mount_nulls: <br />
also goofy mount_nulls seen in df -k on jail2<br><br />
also needs to be done in safe and quad<br><br />
if he wants another, use safe/quad as example and add it to file. also exec command at prompt to add mount (no reboot, remount or rejail necessary)<br />
<br />
== Moving customer to another jail machine ==<br />
<br />
Systems should only be moved between similarly-versioned jails. If you cannot get a match, you should go to freebsd.org and see what the differences were between the version you're moving from and the new version. Many of the updates will be to drivers and kernels - these don't affect the customer. Perhaps they updated traceroute or a library...you should copy the new/changed files into the VPS/jail on the new host. An alternate method for moving to a different version would be to buildworld to bring their full distribution up to date. Optional steps for this kind of upgrade are included below.<br />
<br />
1. <tt>g <customerID></tt><br />
<br />
<br />
2. <tt>[[#jailkill|jailkill]] <hostname></tt><br />
<br />
<br />
3. create new device on target system<br><br />
4.x:<br><br />
<pre>vnconfig –T –S 4g –s labels –c /dev/vn1 /mnt/data1/69.55.22x.x-col00XXX<br />
disklabel –r –w vn1 auto<br />
newfs /dev/vn1c</pre><br />
<br />
6.x:<br><br />
<pre>bsdlabel -r -w /dev/gvinum/v1<br />
newfs /dev/gvinum/v1a<br />
or <br />
gconcat label v1-v3 /dev/gvinum/v1 /dev/gvinum/v2 /dev/gvinum/v3<br />
bsdlabel -r -w /dev/concat/v1-v3<br />
newfs /dev/concat/v1-v3a</pre><br />
<br />
7.x+:<br><br />
Run jailmakeempty then skip step 4 & 7 below<br />
<br />
<br />
4. make and mount the dir on new system. <br><br />
Ex: <pre>mkdir /mnt/data2/69.55.230.3-col00123-DIR<br />
chmod 755 /mnt/data2/69.55.230.3-col00123-DIR<br />
mount [device] /mnt/data2/69.55.230.3-col00123-DIR</pre><br />
<br />
<br />
5. [[#stopjail|stopjail]] <hostname> 1<br />
<br />
<br />
6. dump fs to new system. ex: <tt>[[#dumpremoterestore|dumpremoterestore]] /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR</tt> (make sure you can ssh as root on remote machine) <br />
<br />
<br />
6a. OPTIONAL BUILDWORLD:<br><br />
cd /usr/src<br />
make world DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
cd etc<br />
make distribution DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
(you may have to rm an openssh file and re-make dist)<br />
<br />
rm -rf /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/periodic/daily/400.status-disks<br />
vi /etc/periodic/security/100.chksetuid<br />
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt><br><br />
with: <tt>MP='/' (use single quotes)</tt><br />
<br />
Ask user if they want ports overwritten with current, if yes:<br />
cp -r /usr/ports /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/usr<br />
<br />
on source: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d .<br />
(copy in all info)<br />
<br />
on target: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d . <br />
(paste all info)<br />
<br />
on source: <br />
cat /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group<br />
(copy in all info)<br />
<br />
on target:<br />
cat > /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group <br />
(paste all info)<br />
<br />
<br />
7. edit quad on source system copy over entries for the jail to the target system <br />
take care that the vn/gvinum/gconcat/md devices on the target system are’t in use and that the /mnt/dataN path doesn’t need changing<br />
<br />
<br />
8. run [[#buildsafe|buildsafe]] (if this is <=6.x) on target system (copies changes made to quad into safe file)<br />
<br />
<br />
9. remove ip from source system. Ex: <tt>ipdel 69.55.230.3</tt><br />
<br />
<br />
10. add ip to target system. Ex: <tt>ipadd 69.55.230.3</tt><br />
<br />
<br />
11. start new system, with [[#startjail|startjail]] or manually, pasting entries found from running <tt>g <customerID></tt> on the new system<br />
<br />
<br />
12. run <tt>[[#canceljail|canceljail]] col0xxxx</tt> on source system. This should prompt you to remove backups if any existed. Do not let it do this, rather:<br />
<br />
<br />
13. if backups existed move them to the new host via the <tt>[[#mvbackups_.28freebsd.29|mvbackups]] script</tt><br />
<br />
<br />
14. edit quad on source system, edit comment to reflect a move rather than a cancel, ex: <tt># moved to jail2 col00241</tt><br />
<br />
<br />
15. edit mgmt to reflect new host and dir for new system<br />
<br />
<br />
16. optional add ipfw rules on new system. Ex:<br />
<tt>ipfw add 01231 count ip from 69.55.230.2 to any<br />
ipfw add 01232 count ip from any to 69.55.230.2</tt><br />
<br />
== Increasing inodes for a VPS ==<br />
<br />
when doing the newfs: <br />
newfs -i 4096 ...<br />
<br />
<br />
== Jail services unable to contact services within same jail ==<br />
<br />
This is due to a messed up routing table (only seen it on jail2 before). <br />
<br />
route delete 69.55.228.65/32 route add -net 69.55.228.65/32 -iface fxp0 -nostatic -cloning route add 69.55.228.65 -iface lo0<br />
<br />
<br />
== /dev/null permission resets == <br />
<br />
Applies to 4.x jails only<br />
<br />
For some reason that I do not understand at all, the /dev/null node in customer jails, after the system is restarted, often reverts permissions to 0600 ... which is bad, because non-root processes cannot redirect to /dev/null anymore. In fact, a fair number of server daemons will complain and/or fail if this is the case.<br />
<br />
So, after a restart, after all systems have restarted, run postboot, which does something like:<br />
<br />
for f in `df -k | grep /dev/vn | awk '{print $6}'` ; do chmod 0666 $f/dev/null ; done<br />
<br />
Later, after the system is up, if anyone ever complains about their /dev/null permissions for any reason ... really if anyone mentions<br />
/dev/null in any way, tell them to run:<br />
<br />
chmod 0666 /dev/null<br />
<br />
We have ultimately fixed this by adding chmods to the quad/safe's<br />
<br />
<br />
== mknod in a jail == <br />
<br />
For a customer who wants to run a chroot name server (named) in a jail.<br />
<br />
<pre><br />
cd /mnt/data1/<user directory>/var/named<br />
mknod -c 1 1 null<br />
mknod -c 2 3 random<br />
</pre><br />
<br />
== Postfix Problems ==<br />
<br />
This hasn't happened in a long time, probably no longer valid with modern OS's<br />
<br />
Postfix is an alternate MTA - people replace sendmail with it in much the same way that many people also replace sendmail with qmail. The problem is that if you install postfix inside of a jail, by default it will not work properly - the user will see error messages like this:<br />
<br />
incoming mail generates errors like this:<br />
<br />
mail_queue_enter: create file incoming/298266.55902: File too large<br />
<br />
and further, sending mail generates errors like this:<br />
<br />
postdrop: warning: mail_queue_enter: create file maildrop/627930.56676: File too large<br />
<br />
This is very easy to solve. In fact, the FreeBSD welcome email that is sent out by `jailmake` now contains this block of text:<br />
<br />
- if you plan on installing postfix, email us first and ask for a necessary patch.<br />
<br />
So, anytime you see a support email that says anything about postfix not working, paste that line from the welcome email into your response to them, and right away, forward the postfix patch to them in an email.<br />
<br />
Here are the instructions:<br />
<br />
Simply copy the patch into /usr/ports/mail/postfix/files, rename the file to ‘patch-file_limit.c’ and then cd to /usr/ports/mail/postfix and run `make install`.<br />
<br />
The postfix patch is the very first email in the support email box, and we never delete or save it because we always want it there to forward to people. The instructions in the email are very clear, and it solves their problem 100% of the time.<br />
<br />
<br />
== Problems with `find` ==<br />
<br />
There was a bug in the FreeBSD vn-filesystem code. The bug causes the system to crash or hang when someone runs the `find` command inside their vn-backed filesystem (and even though the bug may have been fixed, due to the i/o overhead, we still don't like find).<br />
<br />
Now, this does not always happen - however, any jail machine with more than 20 systems on it will crash every night if every system on it runs the daily periodic script out of their crontab, and updates their locate database with the find command.<br />
<br />
(the locate database is a small, FreeBSD specific database that is populated nightly from a cron job, and is then used to provide fast answers to the `locate` command)<br />
<br />
In normal systems (32-40 jails on them), all that has to be done is make sure nobody runs the daily periodic. This means that the file /etc/crontab, instead of looking like this:<br />
<br />
<pre># do daily/weekly/monthly maintenance<br />
1 3 * * * root periodic daily<br />
15 4 * * 6 root periodic weekly<br />
30 5 1 * * root periodic monthly</pre><br />
<br />
Needs to look like this:<br />
<br />
<pre># DO NOT UNCOMMENT THESE - contact support@johncompanies.com for details<br />
#1 3 * * * root periodic daily<br />
#15 4 * * 6 root periodic weekly<br />
#30 5 1 * * root periodic monthly<br />
## DO NOT UNCOMMENT THESE ^^^^^^^^</pre><br />
<br />
and the problem will generally not occur at all. New filesystem images are always altered in this way, so all systems are set like this.<br />
<br />
However, if you have enough systems running, even casual uses of find, outside of the periodic scripts, can cause the system to crash. We are seeing this on jail13, which crashes about once per week.<br />
<br />
There is not yet any resolution to this problem.<br />
<br />
So, if you have a freebsd system crash, see if anyone has uncommented their daily periodic line - you can see them for everyone by running:<br />
<br />
/bin/sh<br />
<br />
and then running:<br />
<br />
<pre>for f in `df -k | grep vn | awk '{print $6}'` ; do echo `cat $f/etc/crontab | grep "periodic daily"` $f ; done | more</pre><br />
<br />
(all one line, one command)<br />
<br />
And then edit those /etc/crontabs and comment them out again.<br />
<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
* if you are getting disk full messages for a BSD customer, it's fairly safe to clear out their /usr/ports/distfiles dir<br />
* 4.x: ps and top an only be run by root in these jails. Done on purpose:<br />
As for `ps` and `top` - non-root users can run them, just not successfully because we have locked the permissions on /dev/mem and /dev/kmem to be root-readable only. That is why non-root users cannot successfully run `ps` and `top`.<br />
* user quotas do not work on freebsd jails - you cannot set up quotas at all, and that's that.<br />
* You cannot inject a process into a 4.x jail, only HUP running processes. 6.x and onward you can with jexec<br />
* jails see base machine’s uptime/load when running top/w<br />
* if someone is unable to get in- cant ping, etc- see if they were blocked by castle (DoS), see if their ip is on the system (post reboot it was lost cause wasn’t in the rc.conf). preboot should catch that.<br />
* in FreeBSD you cant su to root unless you belong to wheel group – so if you remove your acct and setup a new one, we have to add it to wheel group (add to /etc/group)<br />
* Dmesg from underlying sys is seen in customers dmesg on jail<br />
* Popper process going crazy @ 40% for 10min = someone who leaves mail on server<br />
* Don’t force umounts on 4.x jails – it crashes the machine, generally ok on newer machines (running md)<br />
* Good book for admin http://search.barnesandnoble.com/booksearch/isbnInquiry.asp?userid=t824VyRAYz&isbn=0596005164&itm=2<br />
* Self-sign ssl cert http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#selfcert<br />
* conversation with Glenn about semaphores and pgsql<br />
<pre>SDBoody: hey, these are valid amounts/figures/increments right:<br />
kern.ipc.semmni=1280<br />
kern.ipc.semmns=1280<br />
<br />
gr8feen: probably... I always forget exactly what those are, so I usually have to look them up<br />
SDBoody: semaphores<br />
SDBoody: i took the current 1024 and added 256 to them<br />
SDBoody: need more for pgsql<br />
gr8feen: I meant the mni and mns parts...<br />
gr8feen: some of those are not ones you just want to add to... hang on a sec and I'll look them up..<br />
gr8feen: what's semmsl set to?<br />
SDBoody: kern.ipc.semmsl: 1024<br />
<br />
SDBoody: kern.ipc.msgseg: 2048 <br />
kern.ipc.msgssz: 8 <br />
kern.ipc.msgtql: 40 <br />
kern.ipc.msgmnb: 2048 <br />
kern.ipc.msgmni: 40 <br />
kern.ipc.msgmax: 16384 <br />
kern.ipc.semaem: 16384 <br />
kern.ipc.semvmx: 65534 <br />
kern.ipc.semusz: 152 <br />
kern.ipc.semume: 10 <br />
kern.ipc.semopm: 100 <br />
kern.ipc.semmsl: 1024 <br />
kern.ipc.semmnu: 512 <br />
kern.ipc.semmns: 1024 <br />
kern.ipc.semmni: 1024 <br />
kern.ipc.semmap: 768 <br />
kern.ipc.shm_allow_removed: 0 <br />
kern.ipc.shm_use_phys: 1 <br />
kern.ipc.shmall: 262144 <br />
kern.ipc.shmseg: 256 <br />
kern.ipc.shmmni: 784 <br />
kern.ipc.shmmin: 1 <br />
kern.ipc.shmmax: 536870912 <br />
kern.ipc.maxsockets: 25600 <br />
<br />
gr8feen: ok...msl is max per id, mni is max ids, mns is max number of semaphores... so you probably want something like mns = mni * msl<br />
gr8feen: which one did you run out of?<br />
SDBoody: not sure how to tell- ipcs shows the sems in use add up to 1024<br />
SDBoody: there are 59 entries<br />
gr8feen: I'm assuming you tried to start postgres and it failed?<br />
SDBoody: yes<br />
gr8feen: it should have logged why, somewhere..<br />
gr8feen: if I recall, it'll tell you which one it ran out of<br />
SDBoody: > DETAIL: Failed system call was semget(1, 17, 03600).<br />
<br />
gr8feen: so it wanted an id with 17 semaphores...I'd start by making mns = 17*mni and leave mni and mnl set to what they are now and see what it does<br />
SDBoody: i think mni is plenty high<br />
SDBoody: ok, more reasonable 17408<br />
gr8feen: yeah...just change that one and see how it goes..<br />
SDBoody: and leave mni alone at 1024?<br />
gr8feen: yeah...mni id the max number of ids...but if it's trying to get something like 17 per id, your going to hit mns before you hit anything else<br />
SDBoody: right, but doesn't hurt to have it that high (assuming)<br />
<br />
gr8feen: not really... I think those get allocated out of ram that you cant page out, but it's still such a small amount that it really doesn't matter<br />
SDBoody: looks like that worked, thx!<br />
gr8feen: cool<br />
<br />
see semaphores: ipcs -a -s</pre><br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
== Making new customer VE (vm) ==<br />
<br />
This applies only to new virts >= 4.x<br />
<br />
grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.<br />
<br />
1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
also, ping the IP address.<br />
<br />
2. put CT on whichever partition has more space<br />
df -h<br />
<br />
3. vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> ) <br />
vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@johncompanies.com centos-6-x86_64 LM<br />
<br />
4. copy veid, dir, ip and password to pending customer screen. activate customer<br />
<br />
== Making new customer VE (vemakexxx) ==<br />
<br />
This applies to older virts with old templates. This should probably not be used at all anymore.<br />
<br />
<br />
1. look thru hist for ip<br />
<br />
2. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
<br />
3. put ve on whichever partition has more space<br />
vemakerh9 <veid> <ip> <hostname> <mount> <email> [gb disk]; <256|384|512> <veid><br />
vemakerh9 866 69.55.226.109 ngentu.com /vz1 ayo@ngantu.com,asd@asd.com 5; 256 866<br />
<br />
4. copy (veid), dir, and ip to pending customer screen (pass set to p455agfa)<br />
<br />
<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrateonline/migrate failures (migrate manually) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
You may also see migration failing due to quota issues.<br />
<br />
You can try to resolve by copying any quota file into the file you need:<br />
<br />
cp /var/vzquota/quota.1 /var/vzquota/quota.xxx<br />
<br />
If it complains about quota running you should then be able to stop it<br />
<br />
vzquota off xxxx<br />
<br />
If all else fails, migrate to a new VEID<br />
i.e. 1234 becomes 12341<br />
<br />
<br />
If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
'''Note:''' don't forget to fix /usr/local/sbin/backup.config for new disk location<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
=== Cant change file /etc/sysconfig/network ===<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
=== What template is user using ===<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
=== Edits needed to run java ===<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
=== Set time zone ===<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
=== Failing shm_open calls ===<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
=== Debian can't get on the network ===<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
=== CentOS can't get on the network ===<br />
<br />
When CentOS system was running, but can't get on the network, they probably updated iproute via yum.<br />
<br />
Copy an older version of the iproute rpm to their VPS.<br />
<br />
<pre>cp -p /root/iproute/iproute-2.6.32-23.el6.<i686>.rpm /vz2/private/<1527>/fs/root</pre><br />
<br />
Enter their ve, install rpm, restart network, and prevent yum from updating iproute.<br />
<br />
<pre><br />
ve <1527><br />
rpm -i --force iproute-2.6.32-23.el6.<i686>.rpm<br />
rpm -q iproute<br />
rpm -e <new iproute rpm><br />
service network restart<br />
echo "exclude=iproute" >>/etc/yum/ip-exclude<br />
echo "include=/etc/yum/ip-exclude" >>/etc/yum.conf<br />
</pre><br />
<br />
<br />
<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
----<br />
<br />
* uptime/load that vz users see isn’t what we see- it's their own load<br />
* Vz overwrites resolve.conf on every boot to what’s in ve config file. Also overwrites hostname<br />
* Quotas in vz: set Gid_limit in vz config file and restart ve. Errors leading up to this will be out of files but no inode or space issues. set to 2000 in conf, and restart vz<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
To create a new Linux VPS<br />
<br />
* get an IP address from mgmt screen<br />
* run df -h to figure out which disk has enough space<br />
* run vm with no arguments to get syntax and exact template name<br />
* use col0xxxx for hostname if they don’t give you a hostname<br />
* copy over dir, veid, ip and password to pending customer screen<br />
<br />
vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> )<br />
<br />
== disable ==<br />
To disable a customers Virtuozzo VPS server.<br />
<br />
* stop the server<br />
vzctl stop <veid><br />
* prevent restart on server reboot and user restart via Control Panel<br />
vzctl set <veid> --disabled=yes --offline_management=no --save<br />
<br />
== cancelve ==<br />
When a customer cancels all service.<br />
<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
<br />
display the contents of the vz config file for the <veid><br />
<br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
<br />
List all virtual environments running on this machine.<br />
<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
<br />
show processes running under this <veid>.<br />
<br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
<br />
Restart a <veid>.<br />
<br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
<br />
enter a <veid> as root.<br />
<br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
<br />
execute a single command in the <veid> environment.<br />
<br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem VEID <256|512|768|1024|1280|1536|2048> [burst GB]<br />
adjusts the memory resources for the VE. If this is on VZ ver >4, you will also be prompted to provide a burst ram amount. Sample usage:<br />
<br />
<pre># setmem 21671 1024 4<br />
/usr/sbin/vzctl set 21671 --ram 1024m --swap 3072m --save<br />
</pre><br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>75.80.128.147https://wiki.jcihosting.com/index.php?title=VPS_Management&diff=2225VPS Management2020-04-30T20:17:10Z<p>75.80.128.147: /* Ganeti Re-Intsall a VPS */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
== Moving a server to a different account in the management system ==<br />
<br />
This is a tough one. You gotta touch a bunch of tables in the mysql database.<br />
<pre><br />
systems<br />
subs<br />
ipmap<br />
</pre><br />
<br />
basically, get the sysid and go off that:<br />
<br />
<pre><br />
update systems set cid='newcid' where sysid=xxxxx<br />
update subs set cid='newcid' where sysid=xxxxx<br />
update ipmap set cid='newcid' where sysid=xxxxx<br />
</pre><br />
<br />
You then need to rename the server disk space to change the old ID to the new one.<br />
<br />
For a jail -<br />
<pre><br />
stopjail <hostname><br />
mv 69.55.2XX.XXX-col0<oldid> 69.55.2XX.XXX-col0<newid><br />
mv 69.55.2XX.XXX-col0<oldid>-DIR 69.55.2XX.XXX-col0<newid>-DIR<br />
vi /usr/local/jail/bin/quad1 [Update the ID in the quad file]<br />
startjail <hostname><br />
</pre><br />
<br />
For a virt -<br />
You will have to <br />
* stop the virt <br />
* rename the directories <br />
* modify the config file /vzconf/<veid>.conf <br />
* rename the virt conf file <br />
* restart the new virt.<br />
<br />
that ought to do it..<br />
<br />
<br />
== Re-enable a customer account in the management system ==<br />
<br />
For example to re-instate col03438, In mysql:<br />
<br />
<pre><br />
show tables ;<br />
select * from customers where cid='col03438' ;<br />
update customers set status=1 where cid='col03438' ;<br />
select * from systems where cid='col03438' ;<br />
update systems set stop_date=NULL where cid='col03438' ;<br />
update systems set cancel_date=NULL where cid='col03438' ;<br />
update systems set status=1 where cid='col03438' ;<br />
select * from subs where cid='col03438' ;<br />
update subs set stop_date=NULL where cid='col03438' ;<br />
update subs set status=1 where cid='col03438' <br />
<br />
</pre><br />
<br />
= Ganeti VPS =<br />
<br />
This is how to create a VPS on the Lamphost ganeti cluster.<br />
I have created scripts for creating a FreeBSD VPS and one for<br />
creating a Linux VPS.<br />
<br />
== Ganeti Install a VPS ==<br />
<br />
To create a linux VPS use<br />
<pre><br />
/root/new <name> <disk size> <memory> <IP Addr> [<OS>]<br />
<br />
( for example /root/new vps.example.com 30 2 69.55.238.10 centos-8 )<br />
</pre><br />
<br />
To create a FreeBSD VPS, we have to do a full install. This will create a <br />
new VPS.<br />
<br />
<pre><br />
/root/newfreebsd12 <name> <disk size> <memory> <IP Addr> [<OS>]<br />
</pre><br />
<br />
You will then have to do the install via VNC.<br />
<br />
You need to customize the install by adding a "jci" account and a customer<br />
account (col0XXXX) as well. <br />
<br />
<br> <br />
You will need to install sudo.<br />
<pre> <br />
pkg install sudo # install sudo package<br />
visudo # to add wheel account to sudoers<br />
vi /etc/group # add jci and user acct to wheel<br />
</pre><br />
<br />
<br><br />
You will also have to set up a console in /etc/ttys.<br />
<pre><br />
vi /etc/ttys<br />
</pre><br />
<br />
== Ganeti shutdown a VPS ==<br />
<br />
To shutdown a ganeti VPS<br />
<br />
<pre><br />
gnt-instance shutdown <VPSname><br />
</pre><br />
<br />
To permanently delete a ganeti VPS<br />
<br />
<pre><br />
gnt-instance remove <VPSname><br />
</pre><br />
<br />
<br />
== Ganeti Re-Intsall a VPS ==<br />
<br />
Re-installing the OS will destroy all data on the VPS.<br />
Login to the Ganeti master node (currently gn1).<br />
<br />
To reinstall the OS on a VPS the VPS must be shutdown.<br />
<pre><br />
gnt-instance shutdown <VPSname><br />
</pre><br />
<br />
Do the reinstall.<br />
<pre><br />
gnt-instance reinstall <VPSname><br />
</pre><br />
<br />
Restart the VPS.<br />
<pre><br />
gnt-instance startup <VPSname><br />
</pre><br />
<br />
Now you need to do the customization for the VPS. This is<br />
basically the same as the Dedicated server customization.<br />
<br />
https://69.55.229.8/index.php/New_Signups#Dedicated_Server_Install_Checklist<br />
<br />
= FreeBSD 10.1 VPS (jail5 and jail6) =<br />
<br />
This is information on how to manage the FreeBSD 10.1 VPS servers on Jail5.<br />
<br />
<br />
== Making new customer VPS ==<br />
<pre><br />
virtmake IP[,IP] <CID> hostname ipfw# email {FSP|FB|FS|FM|FP} [size in GB]<br />
</pre><br />
<br />
Note: <br />
For now, CID must be unique in the last 2 digits of the CID.<br />
<br />
Must do a "vm start <CID>" after creating a VPS.<br />
<br />
== Starting VPS ==<br />
To start all vms on the system<br />
<pre><br />
vm start<br />
</pre><br />
<br />
To start a single VPS <br />
<pre><br />
vm start <CID><br />
</pre><br />
<br />
== List running VPS ==<br />
<pre><br />
vm status<br />
</pre><br />
<br />
== Attach to a user VPS console ==<br />
<pre><br />
vm attach <CID><br />
</pre><br />
<br />
<br />
To detach from a console, type tilde control-d (~ ctl-d).<br />
<br />
== Stop a VPS ==<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
== Restart a VPS ==<br />
<pre><br />
vm restart <CID><br />
</pre><br />
<br />
<br />
== Modify a VPS ==<br />
<br />
To modify a VPS config, stop the VPS<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
Mount the VPS data file<br />
<pre><br />
vm mount <CID><br />
<br />
cd /data/vm/private/<CID>/mount<br />
</pre><br />
<br />
Modify the user files necessary.<br />
<br />
Restart the VPS.<br />
<br />
<pre><br />
vm umount <CID><br />
<br />
vm start <CID><br />
</pre><br />
<br />
== Notes on a new FBSD 10.x VPS (jail6) ==<br />
<br />
<pre><br />
#! /bin/sh<br />
#<br />
# new <colo#> <disk><br />
<br />
<br />
memory=1024<br />
<br />
x="$1"<br />
<br />
disk="$2"<br />
if [ "y" == "y$disk" ] ; then<br />
disk=60<br />
fi<br />
<br />
disk2=`expr $disk - 4`<br />
<br />
<br />
vm image provision 04dcdf96-d843-11e6-b131-001e67fcc64f col0$x<br />
# vm configure col0$x<br />
sed -e "s/memory=256M/memory=${memory}M/" /vms/col01114/col01114.conf >/vms/col0$x/col0$x.conf<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
# vm start col0$x<br />
<br />
<br />
exit<br />
<br />
vm console col0$x<br />
<br />
#<br />
# to increase the customer disk<br />
#<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
<br />
## on the customer VPS<br />
<br />
gpart recover vtbd0<br />
gpart show vtbd0<br />
swapoff /dev/vtbd0p3<br />
gpart delete -i 3 vtbd0<br />
gpart show vtbd0<br />
sysctl kern.geom.debugflags=16<br />
gpart resize -i 2 -a 4k -s $disk2\G vtbd0<br />
<br />
growfs /dev/vtbd0p2<br />
<br />
gpart show vtbd0<br />
gpart add -t freebsd-swap -a 4k vtbd0<br />
gpart show vtbd0<br />
swapon /dev/vtbd0p3<br />
<br />
df -h<br />
reboot<br />
<br />
vi /etc/rc.conf<br />
hostname<br />
IP Addr<br />
passwd root<br />
passwd user<br />
clear history<br />
cp /dev/null /var/log/utx.log<br />
cp /dev/null ~root/.history<br />
history -c<br />
<br />
</pre><br />
<br />
= FreeBSD Jails =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Making new customer jail 4.x ==<br />
<br />
1. run [[#js|js]] to figure out which partition, IP to put it on, find an unused vn, and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vnX hostname shorthost quad/safe# ipfw# email [gb disk, default 4]</tt><br />
<br />
jail14# /tmp/jailmake 69.55.226.152 /mnt/data2/69.55.226.152-col00182 vn23 negev.cerastes.org negev 4 182 cerastes@cerastes.org<br />
<br />
<br />
== Making new customer jail 6.x ==<br />
<br />
1. run [[#js|js]] to figure out which gvinum vols are free (and which mnt (disk) each gvinum goes with/to), IP to put it on and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vN[,vN] hostname shorthost quad/safe# ipfw# email</tt><br />
<br />
jail19# jailmake 69.55.236.17 /mnt/data1/69.55.236.17-col01574 v10 uncle-leo.commadev.com uncle-leo 1 1574 lee@commadev.com,lee@gmail.com<br />
<br />
<br />
== Changing an ip for freebsd VPS ==<br />
<br />
*stopjail (hostname)<br />
*on systems using vnfile or mdfile: rename vnfile/mdfile, using new ip<br />
*rename dir using new ip<br />
*Edit quad (make sure to update all lines: directory, vn/mdfile)<br />
*buildsafe (not necessary on systems having only quad1)<br />
*if you're adding an ip not already on the host machine: <tt>ipadd x.x.x.x</tt><br />
*startjail (hostname)<br />
if backups: <tt>mvbackups</tt> ...<br />
*Edit dir/ip in mgmt<br />
*if any rules, update any firewall rules to use new ip (careful not to make sure there arent multiple rules per ipfw#- search by colo#)<br />
*(if customer asks and has domains) update their domains on ns1c<br />
*optional: update ipfw counters on the host server<br />
<br />
<br />
== Rename a gconcat vol ==<br />
<br />
*stop jail, unmount everything<br />
*gconcat stop vol<br />
*gconcat clear /dev/gvinum/vN<br />
*gconcat clear /dev/gvinum/vN+1<br />
*gconcat label -v newname /dev/gvinum/vN /dev/gvinum/vN+1<br />
*bsdlabel /dev/concat/newname<br />
(make sure a partition is not fstype unused, if so change to: 4.2BSD)<br />
<br />
<br />
== Remaking a VPS (on same jail) ==<br />
<br />
# take him down (stopjail)<br />
# make note of host, vn/gvin/md, ip<br />
# remove from quad/safe<br />
# move vnfile (rename)<br />
# remove dir (jailmake will make a new one)<br />
# remove ipfw counts<br />
# jailmake<br />
# update backup script<br />
# update db with new dir/ip (use the “move” link, mark old as stopped today)<br />
# update firewall if changed ip<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
<br />
== Adding disk to a 4.x jail ==<br />
<br />
NOTE: if you unmount before vnconfig, re-vnconfig then unmount then vnunconfig<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them.<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to dump to a different partition and create the new vn on the current partition. If no space to make make dump, then move old vn’s to backup or worse, dump to backup.<br />
<br />
<br />
2. <tt>g <customerID></tt> to get the info<br />
<br />
<br />
3. make vnfile, newfs and mount. USE CAUTION not to overwrite existing vn. <br><br />
Ex: <tt>vnconfig –T -S 9g -s labels -c /dev/vn22 /mnt/data2/65.214.160.117-col00496<br><br />
disklabel -r -w vn30 auto<br><br />
newfs /dev/vn30c<br><br />
mkdir /mnt/data2/65.214.160.117-col00496-DIR<br><br />
mount /dev/vn30c /mnt/data2/69.55.237.26-col00241-DIR<br><br />
cd /mnt/data2/65.214.160.117-col00496-DIR</tt><br />
<br />
<br />
4. <tt>jailkill <hostname></tt><br />
<br />
<br />
5. <tt>dump -0a -f - /dev/vn1 | restore -r -f -</tt> <br />
<br />
<br />
6. <tt>rm restoresymtable</tt><br />
<br />
<br />
7. unmount and unvnconfig old system:<br><br />
<tt>umount /dev/vnNNc <br><br />
vnconfig –u /dev/vnNN</tt><br />
<br />
<br />
8. move vnfile. Ex: <tt>mv /mnt/data1/69.55.237.26-col00241 /mnt/data1/old-col00241-vnfile-20110312-noarchive</tt><br />
<br />
<br />
9. remove old dir: <tt>rmdir /mnt/data1/69.55.237.26-col00241-DIR</tt><br />
<br />
<br />
10. edit quad/safe to point to new location, run <tt>[[#buildsafe|buildsafe]]</tt><br />
<br />
<br />
11. start jail: <tt>[[#startjail|startjail]] <hostname></tt><br />
<br />
<br />
12. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
13. mv backups if necessary<br />
<br />
WARNING: if you restore to unmounted vn then you’re actually restoring to /mnt/data1<br />
you can tar and do above to recover<br />
<br />
NOTE: you can’t move a customer to a system that has a base sys that is diff from the one they came from (BSD 4.8 to 4.5 = won’t work)<br />
<br />
NOTE: we don’t mount procfs's to anyone’s system by default.<br />
<br />
NOTE: mount_nulls: <br />
also goofy mount_nulls seen in df -k on jail2<br><br />
also needs to be done in safe and quad<br><br />
if he wants another, use safe/quad as example and add it to file. also exec command at prompt to add mount (no reboot, remount or rejail necessary)<br />
<br />
== Moving customer to another jail machine ==<br />
<br />
Systems should only be moved between similarly-versioned jails. If you cannot get a match, you should go to freebsd.org and see what the differences were between the version you're moving from and the new version. Many of the updates will be to drivers and kernels - these don't affect the customer. Perhaps they updated traceroute or a library...you should copy the new/changed files into the VPS/jail on the new host. An alternate method for moving to a different version would be to buildworld to bring their full distribution up to date. Optional steps for this kind of upgrade are included below.<br />
<br />
1. <tt>g <customerID></tt><br />
<br />
<br />
2. <tt>[[#jailkill|jailkill]] <hostname></tt><br />
<br />
<br />
3. create new device on target system<br><br />
4.x:<br><br />
<pre>vnconfig –T –S 4g –s labels –c /dev/vn1 /mnt/data1/69.55.22x.x-col00XXX<br />
disklabel –r –w vn1 auto<br />
newfs /dev/vn1c</pre><br />
<br />
6.x:<br><br />
<pre>bsdlabel -r -w /dev/gvinum/v1<br />
newfs /dev/gvinum/v1a<br />
or <br />
gconcat label v1-v3 /dev/gvinum/v1 /dev/gvinum/v2 /dev/gvinum/v3<br />
bsdlabel -r -w /dev/concat/v1-v3<br />
newfs /dev/concat/v1-v3a</pre><br />
<br />
7.x+:<br><br />
Run jailmakeempty then skip step 4 & 7 below<br />
<br />
<br />
4. make and mount the dir on new system. <br><br />
Ex: <pre>mkdir /mnt/data2/69.55.230.3-col00123-DIR<br />
chmod 755 /mnt/data2/69.55.230.3-col00123-DIR<br />
mount [device] /mnt/data2/69.55.230.3-col00123-DIR</pre><br />
<br />
<br />
5. [[#stopjail|stopjail]] <hostname> 1<br />
<br />
<br />
6. dump fs to new system. ex: <tt>[[#dumpremoterestore|dumpremoterestore]] /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR</tt> (make sure you can ssh as root on remote machine) <br />
<br />
<br />
6a. OPTIONAL BUILDWORLD:<br><br />
cd /usr/src<br />
make world DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
cd etc<br />
make distribution DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
(you may have to rm an openssh file and re-make dist)<br />
<br />
rm -rf /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/periodic/daily/400.status-disks<br />
vi /etc/periodic/security/100.chksetuid<br />
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt><br><br />
with: <tt>MP='/' (use single quotes)</tt><br />
<br />
Ask user if they want ports overwritten with current, if yes:<br />
cp -r /usr/ports /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/usr<br />
<br />
on source: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d .<br />
(copy in all info)<br />
<br />
on target: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d . <br />
(paste all info)<br />
<br />
on source: <br />
cat /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group<br />
(copy in all info)<br />
<br />
on target:<br />
cat > /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group <br />
(paste all info)<br />
<br />
<br />
7. edit quad on source system copy over entries for the jail to the target system <br />
take care that the vn/gvinum/gconcat/md devices on the target system are’t in use and that the /mnt/dataN path doesn’t need changing<br />
<br />
<br />
8. run [[#buildsafe|buildsafe]] (if this is <=6.x) on target system (copies changes made to quad into safe file)<br />
<br />
<br />
9. remove ip from source system. Ex: <tt>ipdel 69.55.230.3</tt><br />
<br />
<br />
10. add ip to target system. Ex: <tt>ipadd 69.55.230.3</tt><br />
<br />
<br />
11. start new system, with [[#startjail|startjail]] or manually, pasting entries found from running <tt>g <customerID></tt> on the new system<br />
<br />
<br />
12. run <tt>[[#canceljail|canceljail]] col0xxxx</tt> on source system. This should prompt you to remove backups if any existed. Do not let it do this, rather:<br />
<br />
<br />
13. if backups existed move them to the new host via the <tt>[[#mvbackups_.28freebsd.29|mvbackups]] script</tt><br />
<br />
<br />
14. edit quad on source system, edit comment to reflect a move rather than a cancel, ex: <tt># moved to jail2 col00241</tt><br />
<br />
<br />
15. edit mgmt to reflect new host and dir for new system<br />
<br />
<br />
16. optional add ipfw rules on new system. Ex:<br />
<tt>ipfw add 01231 count ip from 69.55.230.2 to any<br />
ipfw add 01232 count ip from any to 69.55.230.2</tt><br />
<br />
== Increasing inodes for a VPS ==<br />
<br />
when doing the newfs: <br />
newfs -i 4096 ...<br />
<br />
<br />
== Jail services unable to contact services within same jail ==<br />
<br />
This is due to a messed up routing table (only seen it on jail2 before). <br />
<br />
route delete 69.55.228.65/32 route add -net 69.55.228.65/32 -iface fxp0 -nostatic -cloning route add 69.55.228.65 -iface lo0<br />
<br />
<br />
== /dev/null permission resets == <br />
<br />
Applies to 4.x jails only<br />
<br />
For some reason that I do not understand at all, the /dev/null node in customer jails, after the system is restarted, often reverts permissions to 0600 ... which is bad, because non-root processes cannot redirect to /dev/null anymore. In fact, a fair number of server daemons will complain and/or fail if this is the case.<br />
<br />
So, after a restart, after all systems have restarted, run postboot, which does something like:<br />
<br />
for f in `df -k | grep /dev/vn | awk '{print $6}'` ; do chmod 0666 $f/dev/null ; done<br />
<br />
Later, after the system is up, if anyone ever complains about their /dev/null permissions for any reason ... really if anyone mentions<br />
/dev/null in any way, tell them to run:<br />
<br />
chmod 0666 /dev/null<br />
<br />
We have ultimately fixed this by adding chmods to the quad/safe's<br />
<br />
<br />
== mknod in a jail == <br />
<br />
For a customer who wants to run a chroot name server (named) in a jail.<br />
<br />
<pre><br />
cd /mnt/data1/<user directory>/var/named<br />
mknod -c 1 1 null<br />
mknod -c 2 3 random<br />
</pre><br />
<br />
== Postfix Problems ==<br />
<br />
This hasn't happened in a long time, probably no longer valid with modern OS's<br />
<br />
Postfix is an alternate MTA - people replace sendmail with it in much the same way that many people also replace sendmail with qmail. The problem is that if you install postfix inside of a jail, by default it will not work properly - the user will see error messages like this:<br />
<br />
incoming mail generates errors like this:<br />
<br />
mail_queue_enter: create file incoming/298266.55902: File too large<br />
<br />
and further, sending mail generates errors like this:<br />
<br />
postdrop: warning: mail_queue_enter: create file maildrop/627930.56676: File too large<br />
<br />
This is very easy to solve. In fact, the FreeBSD welcome email that is sent out by `jailmake` now contains this block of text:<br />
<br />
- if you plan on installing postfix, email us first and ask for a necessary patch.<br />
<br />
So, anytime you see a support email that says anything about postfix not working, paste that line from the welcome email into your response to them, and right away, forward the postfix patch to them in an email.<br />
<br />
Here are the instructions:<br />
<br />
Simply copy the patch into /usr/ports/mail/postfix/files, rename the file to ‘patch-file_limit.c’ and then cd to /usr/ports/mail/postfix and run `make install`.<br />
<br />
The postfix patch is the very first email in the support email box, and we never delete or save it because we always want it there to forward to people. The instructions in the email are very clear, and it solves their problem 100% of the time.<br />
<br />
<br />
== Problems with `find` ==<br />
<br />
There was a bug in the FreeBSD vn-filesystem code. The bug causes the system to crash or hang when someone runs the `find` command inside their vn-backed filesystem (and even though the bug may have been fixed, due to the i/o overhead, we still don't like find).<br />
<br />
Now, this does not always happen - however, any jail machine with more than 20 systems on it will crash every night if every system on it runs the daily periodic script out of their crontab, and updates their locate database with the find command.<br />
<br />
(the locate database is a small, FreeBSD specific database that is populated nightly from a cron job, and is then used to provide fast answers to the `locate` command)<br />
<br />
In normal systems (32-40 jails on them), all that has to be done is make sure nobody runs the daily periodic. This means that the file /etc/crontab, instead of looking like this:<br />
<br />
<pre># do daily/weekly/monthly maintenance<br />
1 3 * * * root periodic daily<br />
15 4 * * 6 root periodic weekly<br />
30 5 1 * * root periodic monthly</pre><br />
<br />
Needs to look like this:<br />
<br />
<pre># DO NOT UNCOMMENT THESE - contact support@johncompanies.com for details<br />
#1 3 * * * root periodic daily<br />
#15 4 * * 6 root periodic weekly<br />
#30 5 1 * * root periodic monthly<br />
## DO NOT UNCOMMENT THESE ^^^^^^^^</pre><br />
<br />
and the problem will generally not occur at all. New filesystem images are always altered in this way, so all systems are set like this.<br />
<br />
However, if you have enough systems running, even casual uses of find, outside of the periodic scripts, can cause the system to crash. We are seeing this on jail13, which crashes about once per week.<br />
<br />
There is not yet any resolution to this problem.<br />
<br />
So, if you have a freebsd system crash, see if anyone has uncommented their daily periodic line - you can see them for everyone by running:<br />
<br />
/bin/sh<br />
<br />
and then running:<br />
<br />
<pre>for f in `df -k | grep vn | awk '{print $6}'` ; do echo `cat $f/etc/crontab | grep "periodic daily"` $f ; done | more</pre><br />
<br />
(all one line, one command)<br />
<br />
And then edit those /etc/crontabs and comment them out again.<br />
<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
* if you are getting disk full messages for a BSD customer, it's fairly safe to clear out their /usr/ports/distfiles dir<br />
* 4.x: ps and top an only be run by root in these jails. Done on purpose:<br />
As for `ps` and `top` - non-root users can run them, just not successfully because we have locked the permissions on /dev/mem and /dev/kmem to be root-readable only. That is why non-root users cannot successfully run `ps` and `top`.<br />
* user quotas do not work on freebsd jails - you cannot set up quotas at all, and that's that.<br />
* You cannot inject a process into a 4.x jail, only HUP running processes. 6.x and onward you can with jexec<br />
* jails see base machine’s uptime/load when running top/w<br />
* if someone is unable to get in- cant ping, etc- see if they were blocked by castle (DoS), see if their ip is on the system (post reboot it was lost cause wasn’t in the rc.conf). preboot should catch that.<br />
* in FreeBSD you cant su to root unless you belong to wheel group – so if you remove your acct and setup a new one, we have to add it to wheel group (add to /etc/group)<br />
* Dmesg from underlying sys is seen in customers dmesg on jail<br />
* Popper process going crazy @ 40% for 10min = someone who leaves mail on server<br />
* Don’t force umounts on 4.x jails – it crashes the machine, generally ok on newer machines (running md)<br />
* Good book for admin http://search.barnesandnoble.com/booksearch/isbnInquiry.asp?userid=t824VyRAYz&isbn=0596005164&itm=2<br />
* Self-sign ssl cert http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#selfcert<br />
* conversation with Glenn about semaphores and pgsql<br />
<pre>SDBoody: hey, these are valid amounts/figures/increments right:<br />
kern.ipc.semmni=1280<br />
kern.ipc.semmns=1280<br />
<br />
gr8feen: probably... I always forget exactly what those are, so I usually have to look them up<br />
SDBoody: semaphores<br />
SDBoody: i took the current 1024 and added 256 to them<br />
SDBoody: need more for pgsql<br />
gr8feen: I meant the mni and mns parts...<br />
gr8feen: some of those are not ones you just want to add to... hang on a sec and I'll look them up..<br />
gr8feen: what's semmsl set to?<br />
SDBoody: kern.ipc.semmsl: 1024<br />
<br />
SDBoody: kern.ipc.msgseg: 2048 <br />
kern.ipc.msgssz: 8 <br />
kern.ipc.msgtql: 40 <br />
kern.ipc.msgmnb: 2048 <br />
kern.ipc.msgmni: 40 <br />
kern.ipc.msgmax: 16384 <br />
kern.ipc.semaem: 16384 <br />
kern.ipc.semvmx: 65534 <br />
kern.ipc.semusz: 152 <br />
kern.ipc.semume: 10 <br />
kern.ipc.semopm: 100 <br />
kern.ipc.semmsl: 1024 <br />
kern.ipc.semmnu: 512 <br />
kern.ipc.semmns: 1024 <br />
kern.ipc.semmni: 1024 <br />
kern.ipc.semmap: 768 <br />
kern.ipc.shm_allow_removed: 0 <br />
kern.ipc.shm_use_phys: 1 <br />
kern.ipc.shmall: 262144 <br />
kern.ipc.shmseg: 256 <br />
kern.ipc.shmmni: 784 <br />
kern.ipc.shmmin: 1 <br />
kern.ipc.shmmax: 536870912 <br />
kern.ipc.maxsockets: 25600 <br />
<br />
gr8feen: ok...msl is max per id, mni is max ids, mns is max number of semaphores... so you probably want something like mns = mni * msl<br />
gr8feen: which one did you run out of?<br />
SDBoody: not sure how to tell- ipcs shows the sems in use add up to 1024<br />
SDBoody: there are 59 entries<br />
gr8feen: I'm assuming you tried to start postgres and it failed?<br />
SDBoody: yes<br />
gr8feen: it should have logged why, somewhere..<br />
gr8feen: if I recall, it'll tell you which one it ran out of<br />
SDBoody: > DETAIL: Failed system call was semget(1, 17, 03600).<br />
<br />
gr8feen: so it wanted an id with 17 semaphores...I'd start by making mns = 17*mni and leave mni and mnl set to what they are now and see what it does<br />
SDBoody: i think mni is plenty high<br />
SDBoody: ok, more reasonable 17408<br />
gr8feen: yeah...just change that one and see how it goes..<br />
SDBoody: and leave mni alone at 1024?<br />
gr8feen: yeah...mni id the max number of ids...but if it's trying to get something like 17 per id, your going to hit mns before you hit anything else<br />
SDBoody: right, but doesn't hurt to have it that high (assuming)<br />
<br />
gr8feen: not really... I think those get allocated out of ram that you cant page out, but it's still such a small amount that it really doesn't matter<br />
SDBoody: looks like that worked, thx!<br />
gr8feen: cool<br />
<br />
see semaphores: ipcs -a -s</pre><br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
== Making new customer VE (vm) ==<br />
<br />
This applies only to new virts >= 4.x<br />
<br />
grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.<br />
<br />
1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
also, ping the IP address.<br />
<br />
2. put CT on whichever partition has more space<br />
df -h<br />
<br />
3. vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> ) <br />
vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@johncompanies.com centos-6-x86_64 LM<br />
<br />
4. copy veid, dir, ip and password to pending customer screen. activate customer<br />
<br />
== Making new customer VE (vemakexxx) ==<br />
<br />
This applies to older virts with old templates. This should probably not be used at all anymore.<br />
<br />
<br />
1. look thru hist for ip<br />
<br />
2. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
<br />
3. put ve on whichever partition has more space<br />
vemakerh9 <veid> <ip> <hostname> <mount> <email> [gb disk]; <256|384|512> <veid><br />
vemakerh9 866 69.55.226.109 ngentu.com /vz1 ayo@ngantu.com,asd@asd.com 5; 256 866<br />
<br />
4. copy (veid), dir, and ip to pending customer screen (pass set to p455agfa)<br />
<br />
<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrateonline/migrate failures (migrate manually) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
You may also see migration failing due to quota issues.<br />
<br />
You can try to resolve by copying any quota file into the file you need:<br />
<br />
cp /var/vzquota/quota.1 /var/vzquota/quota.xxx<br />
<br />
If it complains about quota running you should then be able to stop it<br />
<br />
vzquota off xxxx<br />
<br />
If all else fails, migrate to a new VEID<br />
i.e. 1234 becomes 12341<br />
<br />
<br />
If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
'''Note:''' don't forget to fix /usr/local/sbin/backup.config for new disk location<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
=== Cant change file /etc/sysconfig/network ===<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
=== What template is user using ===<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
=== Edits needed to run java ===<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
=== Set time zone ===<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
=== Failing shm_open calls ===<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
=== Debian can't get on the network ===<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
=== CentOS can't get on the network ===<br />
<br />
When CentOS system was running, but can't get on the network, they probably updated iproute via yum.<br />
<br />
Copy an older version of the iproute rpm to their VPS.<br />
<br />
<pre>cp -p /root/iproute/iproute-2.6.32-23.el6.<i686>.rpm /vz2/private/<1527>/fs/root</pre><br />
<br />
Enter their ve, install rpm, restart network, and prevent yum from updating iproute.<br />
<br />
<pre><br />
ve <1527><br />
rpm -i --force iproute-2.6.32-23.el6.<i686>.rpm<br />
rpm -q iproute<br />
rpm -e <new iproute rpm><br />
service network restart<br />
echo "exclude=iproute" >>/etc/yum/ip-exclude<br />
echo "include=/etc/yum/ip-exclude" >>/etc/yum.conf<br />
</pre><br />
<br />
<br />
<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
----<br />
<br />
* uptime/load that vz users see isn’t what we see- it's their own load<br />
* Vz overwrites resolve.conf on every boot to what’s in ve config file. Also overwrites hostname<br />
* Quotas in vz: set Gid_limit in vz config file and restart ve. Errors leading up to this will be out of files but no inode or space issues. set to 2000 in conf, and restart vz<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
To create a new Linux VPS<br />
<br />
* get an IP address from mgmt screen<br />
* run df -h to figure out which disk has enough space<br />
* run vm with no arguments to get syntax and exact template name<br />
* use col0xxxx for hostname if they don’t give you a hostname<br />
* copy over dir, veid, ip and password to pending customer screen<br />
<br />
vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> )<br />
<br />
== disable ==<br />
To disable a customers Virtuozzo VPS server.<br />
<br />
* stop the server<br />
vzctl stop <veid><br />
* prevent restart on server reboot and user restart via Control Panel<br />
vzctl set <veid> --disabled=yes --offline_management=no --save<br />
<br />
== cancelve ==<br />
When a customer cancels all service.<br />
<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
<br />
display the contents of the vz config file for the <veid><br />
<br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
<br />
List all virtual environments running on this machine.<br />
<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
<br />
show processes running under this <veid>.<br />
<br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
<br />
Restart a <veid>.<br />
<br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
<br />
enter a <veid> as root.<br />
<br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
<br />
execute a single command in the <veid> environment.<br />
<br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem VEID <256|512|768|1024|1280|1536|2048> [burst GB]<br />
adjusts the memory resources for the VE. If this is on VZ ver >4, you will also be prompted to provide a burst ram amount. Sample usage:<br />
<br />
<pre># setmem 21671 1024 4<br />
/usr/sbin/vzctl set 21671 --ram 1024m --swap 3072m --save<br />
</pre><br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>75.80.128.147https://wiki.jcihosting.com/index.php?title=VPS_Management&diff=2224VPS Management2020-04-03T17:25:57Z<p>75.80.128.147: /* Ganeti Re-Insall a VPS */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
== Moving a server to a different account in the management system ==<br />
<br />
This is a tough one. You gotta touch a bunch of tables in the mysql database.<br />
<pre><br />
systems<br />
subs<br />
ipmap<br />
</pre><br />
<br />
basically, get the sysid and go off that:<br />
<br />
<pre><br />
update systems set cid='newcid' where sysid=xxxxx<br />
update subs set cid='newcid' where sysid=xxxxx<br />
update ipmap set cid='newcid' where sysid=xxxxx<br />
</pre><br />
<br />
You then need to rename the server disk space to change the old ID to the new one.<br />
<br />
For a jail -<br />
<pre><br />
stopjail <hostname><br />
mv 69.55.2XX.XXX-col0<oldid> 69.55.2XX.XXX-col0<newid><br />
mv 69.55.2XX.XXX-col0<oldid>-DIR 69.55.2XX.XXX-col0<newid>-DIR<br />
vi /usr/local/jail/bin/quad1 [Update the ID in the quad file]<br />
startjail <hostname><br />
</pre><br />
<br />
For a virt -<br />
You will have to <br />
* stop the virt <br />
* rename the directories <br />
* modify the config file /vzconf/<veid>.conf <br />
* rename the virt conf file <br />
* restart the new virt.<br />
<br />
that ought to do it..<br />
<br />
<br />
== Re-enable a customer account in the management system ==<br />
<br />
For example to re-instate col03438, In mysql:<br />
<br />
<pre><br />
show tables ;<br />
select * from customers where cid='col03438' ;<br />
update customers set status=1 where cid='col03438' ;<br />
select * from systems where cid='col03438' ;<br />
update systems set stop_date=NULL where cid='col03438' ;<br />
update systems set cancel_date=NULL where cid='col03438' ;<br />
update systems set status=1 where cid='col03438' ;<br />
select * from subs where cid='col03438' ;<br />
update subs set stop_date=NULL where cid='col03438' ;<br />
update subs set status=1 where cid='col03438' <br />
<br />
</pre><br />
<br />
= Ganeti VPS =<br />
<br />
This is how to create a VPS on the Lamphost ganeti cluster.<br />
I have created scripts for creating a FreeBSD VPS and one for<br />
creating a Linux VPS.<br />
<br />
== Ganeti Install a VPS ==<br />
<br />
To create a linux VPS use<br />
<pre><br />
/root/new <name> <disk size> <memory> <IP Addr> [<OS>]<br />
<br />
( for example /root/new vps.example.com 30 2 69.55.238.10 centos-8 )<br />
</pre><br />
<br />
To create a FreeBSD VPS, we have to do a full install. This will create a <br />
new VPS.<br />
<br />
<pre><br />
/root/newfreebsd12 <name> <disk size> <memory> <IP Addr> [<OS>]<br />
</pre><br />
<br />
You will then have to do the install via VNC.<br />
<br />
You need to customize the install by adding a "jci" account and a customer<br />
account (col0XXXX) as well. <br />
<br />
<br> <br />
You will need to install sudo.<br />
<pre> <br />
pkg install sudo # install sudo package<br />
visudo # to add wheel account to sudoers<br />
vi /etc/group # add jci and user acct to wheel<br />
</pre><br />
<br />
<br><br />
You will also have to set up a console in /etc/ttys.<br />
<pre><br />
vi /etc/ttys<br />
</pre><br />
<br />
== Ganeti Re-Intsall a VPS ==<br />
<br />
Re-installing the OS will destroy all data on the VPS.<br />
Login to the Ganeti master node (currently gn1).<br />
<br />
To reinstall the OS on a VPS the VPS must be shutdown.<br />
<pre><br />
gnt-instance shutdown <VPSname><br />
</pre><br />
<br />
Do the reinstall.<br />
<pre><br />
gnt-instance reinstall <VPSname><br />
</pre><br />
<br />
Restart the VPS.<br />
<pre><br />
gnt-instance startup <VPSname><br />
</pre><br />
<br />
Now you need to do the customization for the VPS. This is<br />
basically the same as the Dedicated server customization.<br />
<br />
https://69.55.229.8/index.php/New_Signups#Dedicated_Server_Install_Checklist<br />
<br />
= FreeBSD 10.1 VPS (jail5 and jail6) =<br />
<br />
This is information on how to manage the FreeBSD 10.1 VPS servers on Jail5.<br />
<br />
<br />
== Making new customer VPS ==<br />
<pre><br />
virtmake IP[,IP] <CID> hostname ipfw# email {FSP|FB|FS|FM|FP} [size in GB]<br />
</pre><br />
<br />
Note: <br />
For now, CID must be unique in the last 2 digits of the CID.<br />
<br />
Must do a "vm start <CID>" after creating a VPS.<br />
<br />
== Starting VPS ==<br />
To start all vms on the system<br />
<pre><br />
vm start<br />
</pre><br />
<br />
To start a single VPS <br />
<pre><br />
vm start <CID><br />
</pre><br />
<br />
== List running VPS ==<br />
<pre><br />
vm status<br />
</pre><br />
<br />
== Attach to a user VPS console ==<br />
<pre><br />
vm attach <CID><br />
</pre><br />
<br />
<br />
To detach from a console, type tilde control-d (~ ctl-d).<br />
<br />
== Stop a VPS ==<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
== Restart a VPS ==<br />
<pre><br />
vm restart <CID><br />
</pre><br />
<br />
<br />
== Modify a VPS ==<br />
<br />
To modify a VPS config, stop the VPS<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
Mount the VPS data file<br />
<pre><br />
vm mount <CID><br />
<br />
cd /data/vm/private/<CID>/mount<br />
</pre><br />
<br />
Modify the user files necessary.<br />
<br />
Restart the VPS.<br />
<br />
<pre><br />
vm umount <CID><br />
<br />
vm start <CID><br />
</pre><br />
<br />
== Notes on a new FBSD 10.x VPS (jail6) ==<br />
<br />
<pre><br />
#! /bin/sh<br />
#<br />
# new <colo#> <disk><br />
<br />
<br />
memory=1024<br />
<br />
x="$1"<br />
<br />
disk="$2"<br />
if [ "y" == "y$disk" ] ; then<br />
disk=60<br />
fi<br />
<br />
disk2=`expr $disk - 4`<br />
<br />
<br />
vm image provision 04dcdf96-d843-11e6-b131-001e67fcc64f col0$x<br />
# vm configure col0$x<br />
sed -e "s/memory=256M/memory=${memory}M/" /vms/col01114/col01114.conf >/vms/col0$x/col0$x.conf<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
# vm start col0$x<br />
<br />
<br />
exit<br />
<br />
vm console col0$x<br />
<br />
#<br />
# to increase the customer disk<br />
#<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
<br />
## on the customer VPS<br />
<br />
gpart recover vtbd0<br />
gpart show vtbd0<br />
swapoff /dev/vtbd0p3<br />
gpart delete -i 3 vtbd0<br />
gpart show vtbd0<br />
sysctl kern.geom.debugflags=16<br />
gpart resize -i 2 -a 4k -s $disk2\G vtbd0<br />
<br />
growfs /dev/vtbd0p2<br />
<br />
gpart show vtbd0<br />
gpart add -t freebsd-swap -a 4k vtbd0<br />
gpart show vtbd0<br />
swapon /dev/vtbd0p3<br />
<br />
df -h<br />
reboot<br />
<br />
vi /etc/rc.conf<br />
hostname<br />
IP Addr<br />
passwd root<br />
passwd user<br />
clear history<br />
cp /dev/null /var/log/utx.log<br />
cp /dev/null ~root/.history<br />
history -c<br />
<br />
</pre><br />
<br />
= FreeBSD Jails =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Making new customer jail 4.x ==<br />
<br />
1. run [[#js|js]] to figure out which partition, IP to put it on, find an unused vn, and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vnX hostname shorthost quad/safe# ipfw# email [gb disk, default 4]</tt><br />
<br />
jail14# /tmp/jailmake 69.55.226.152 /mnt/data2/69.55.226.152-col00182 vn23 negev.cerastes.org negev 4 182 cerastes@cerastes.org<br />
<br />
<br />
== Making new customer jail 6.x ==<br />
<br />
1. run [[#js|js]] to figure out which gvinum vols are free (and which mnt (disk) each gvinum goes with/to), IP to put it on and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vN[,vN] hostname shorthost quad/safe# ipfw# email</tt><br />
<br />
jail19# jailmake 69.55.236.17 /mnt/data1/69.55.236.17-col01574 v10 uncle-leo.commadev.com uncle-leo 1 1574 lee@commadev.com,lee@gmail.com<br />
<br />
<br />
== Changing an ip for freebsd VPS ==<br />
<br />
*stopjail (hostname)<br />
*on systems using vnfile or mdfile: rename vnfile/mdfile, using new ip<br />
*rename dir using new ip<br />
*Edit quad (make sure to update all lines: directory, vn/mdfile)<br />
*buildsafe (not necessary on systems having only quad1)<br />
*if you're adding an ip not already on the host machine: <tt>ipadd x.x.x.x</tt><br />
*startjail (hostname)<br />
if backups: <tt>mvbackups</tt> ...<br />
*Edit dir/ip in mgmt<br />
*if any rules, update any firewall rules to use new ip (careful not to make sure there arent multiple rules per ipfw#- search by colo#)<br />
*(if customer asks and has domains) update their domains on ns1c<br />
*optional: update ipfw counters on the host server<br />
<br />
<br />
== Rename a gconcat vol ==<br />
<br />
*stop jail, unmount everything<br />
*gconcat stop vol<br />
*gconcat clear /dev/gvinum/vN<br />
*gconcat clear /dev/gvinum/vN+1<br />
*gconcat label -v newname /dev/gvinum/vN /dev/gvinum/vN+1<br />
*bsdlabel /dev/concat/newname<br />
(make sure a partition is not fstype unused, if so change to: 4.2BSD)<br />
<br />
<br />
== Remaking a VPS (on same jail) ==<br />
<br />
# take him down (stopjail)<br />
# make note of host, vn/gvin/md, ip<br />
# remove from quad/safe<br />
# move vnfile (rename)<br />
# remove dir (jailmake will make a new one)<br />
# remove ipfw counts<br />
# jailmake<br />
# update backup script<br />
# update db with new dir/ip (use the “move” link, mark old as stopped today)<br />
# update firewall if changed ip<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
<br />
== Adding disk to a 4.x jail ==<br />
<br />
NOTE: if you unmount before vnconfig, re-vnconfig then unmount then vnunconfig<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them.<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to dump to a different partition and create the new vn on the current partition. If no space to make make dump, then move old vn’s to backup or worse, dump to backup.<br />
<br />
<br />
2. <tt>g <customerID></tt> to get the info<br />
<br />
<br />
3. make vnfile, newfs and mount. USE CAUTION not to overwrite existing vn. <br><br />
Ex: <tt>vnconfig –T -S 9g -s labels -c /dev/vn22 /mnt/data2/65.214.160.117-col00496<br><br />
disklabel -r -w vn30 auto<br><br />
newfs /dev/vn30c<br><br />
mkdir /mnt/data2/65.214.160.117-col00496-DIR<br><br />
mount /dev/vn30c /mnt/data2/69.55.237.26-col00241-DIR<br><br />
cd /mnt/data2/65.214.160.117-col00496-DIR</tt><br />
<br />
<br />
4. <tt>jailkill <hostname></tt><br />
<br />
<br />
5. <tt>dump -0a -f - /dev/vn1 | restore -r -f -</tt> <br />
<br />
<br />
6. <tt>rm restoresymtable</tt><br />
<br />
<br />
7. unmount and unvnconfig old system:<br><br />
<tt>umount /dev/vnNNc <br><br />
vnconfig –u /dev/vnNN</tt><br />
<br />
<br />
8. move vnfile. Ex: <tt>mv /mnt/data1/69.55.237.26-col00241 /mnt/data1/old-col00241-vnfile-20110312-noarchive</tt><br />
<br />
<br />
9. remove old dir: <tt>rmdir /mnt/data1/69.55.237.26-col00241-DIR</tt><br />
<br />
<br />
10. edit quad/safe to point to new location, run <tt>[[#buildsafe|buildsafe]]</tt><br />
<br />
<br />
11. start jail: <tt>[[#startjail|startjail]] <hostname></tt><br />
<br />
<br />
12. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
13. mv backups if necessary<br />
<br />
WARNING: if you restore to unmounted vn then you’re actually restoring to /mnt/data1<br />
you can tar and do above to recover<br />
<br />
NOTE: you can’t move a customer to a system that has a base sys that is diff from the one they came from (BSD 4.8 to 4.5 = won’t work)<br />
<br />
NOTE: we don’t mount procfs's to anyone’s system by default.<br />
<br />
NOTE: mount_nulls: <br />
also goofy mount_nulls seen in df -k on jail2<br><br />
also needs to be done in safe and quad<br><br />
if he wants another, use safe/quad as example and add it to file. also exec command at prompt to add mount (no reboot, remount or rejail necessary)<br />
<br />
== Moving customer to another jail machine ==<br />
<br />
Systems should only be moved between similarly-versioned jails. If you cannot get a match, you should go to freebsd.org and see what the differences were between the version you're moving from and the new version. Many of the updates will be to drivers and kernels - these don't affect the customer. Perhaps they updated traceroute or a library...you should copy the new/changed files into the VPS/jail on the new host. An alternate method for moving to a different version would be to buildworld to bring their full distribution up to date. Optional steps for this kind of upgrade are included below.<br />
<br />
1. <tt>g <customerID></tt><br />
<br />
<br />
2. <tt>[[#jailkill|jailkill]] <hostname></tt><br />
<br />
<br />
3. create new device on target system<br><br />
4.x:<br><br />
<pre>vnconfig –T –S 4g –s labels –c /dev/vn1 /mnt/data1/69.55.22x.x-col00XXX<br />
disklabel –r –w vn1 auto<br />
newfs /dev/vn1c</pre><br />
<br />
6.x:<br><br />
<pre>bsdlabel -r -w /dev/gvinum/v1<br />
newfs /dev/gvinum/v1a<br />
or <br />
gconcat label v1-v3 /dev/gvinum/v1 /dev/gvinum/v2 /dev/gvinum/v3<br />
bsdlabel -r -w /dev/concat/v1-v3<br />
newfs /dev/concat/v1-v3a</pre><br />
<br />
7.x+:<br><br />
Run jailmakeempty then skip step 4 & 7 below<br />
<br />
<br />
4. make and mount the dir on new system. <br><br />
Ex: <pre>mkdir /mnt/data2/69.55.230.3-col00123-DIR<br />
chmod 755 /mnt/data2/69.55.230.3-col00123-DIR<br />
mount [device] /mnt/data2/69.55.230.3-col00123-DIR</pre><br />
<br />
<br />
5. [[#stopjail|stopjail]] <hostname> 1<br />
<br />
<br />
6. dump fs to new system. ex: <tt>[[#dumpremoterestore|dumpremoterestore]] /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR</tt> (make sure you can ssh as root on remote machine) <br />
<br />
<br />
6a. OPTIONAL BUILDWORLD:<br><br />
cd /usr/src<br />
make world DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
cd etc<br />
make distribution DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
(you may have to rm an openssh file and re-make dist)<br />
<br />
rm -rf /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/periodic/daily/400.status-disks<br />
vi /etc/periodic/security/100.chksetuid<br />
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt><br><br />
with: <tt>MP='/' (use single quotes)</tt><br />
<br />
Ask user if they want ports overwritten with current, if yes:<br />
cp -r /usr/ports /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/usr<br />
<br />
on source: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d .<br />
(copy in all info)<br />
<br />
on target: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d . <br />
(paste all info)<br />
<br />
on source: <br />
cat /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group<br />
(copy in all info)<br />
<br />
on target:<br />
cat > /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group <br />
(paste all info)<br />
<br />
<br />
7. edit quad on source system copy over entries for the jail to the target system <br />
take care that the vn/gvinum/gconcat/md devices on the target system are’t in use and that the /mnt/dataN path doesn’t need changing<br />
<br />
<br />
8. run [[#buildsafe|buildsafe]] (if this is <=6.x) on target system (copies changes made to quad into safe file)<br />
<br />
<br />
9. remove ip from source system. Ex: <tt>ipdel 69.55.230.3</tt><br />
<br />
<br />
10. add ip to target system. Ex: <tt>ipadd 69.55.230.3</tt><br />
<br />
<br />
11. start new system, with [[#startjail|startjail]] or manually, pasting entries found from running <tt>g <customerID></tt> on the new system<br />
<br />
<br />
12. run <tt>[[#canceljail|canceljail]] col0xxxx</tt> on source system. This should prompt you to remove backups if any existed. Do not let it do this, rather:<br />
<br />
<br />
13. if backups existed move them to the new host via the <tt>[[#mvbackups_.28freebsd.29|mvbackups]] script</tt><br />
<br />
<br />
14. edit quad on source system, edit comment to reflect a move rather than a cancel, ex: <tt># moved to jail2 col00241</tt><br />
<br />
<br />
15. edit mgmt to reflect new host and dir for new system<br />
<br />
<br />
16. optional add ipfw rules on new system. Ex:<br />
<tt>ipfw add 01231 count ip from 69.55.230.2 to any<br />
ipfw add 01232 count ip from any to 69.55.230.2</tt><br />
<br />
== Increasing inodes for a VPS ==<br />
<br />
when doing the newfs: <br />
newfs -i 4096 ...<br />
<br />
<br />
== Jail services unable to contact services within same jail ==<br />
<br />
This is due to a messed up routing table (only seen it on jail2 before). <br />
<br />
route delete 69.55.228.65/32 route add -net 69.55.228.65/32 -iface fxp0 -nostatic -cloning route add 69.55.228.65 -iface lo0<br />
<br />
<br />
== /dev/null permission resets == <br />
<br />
Applies to 4.x jails only<br />
<br />
For some reason that I do not understand at all, the /dev/null node in customer jails, after the system is restarted, often reverts permissions to 0600 ... which is bad, because non-root processes cannot redirect to /dev/null anymore. In fact, a fair number of server daemons will complain and/or fail if this is the case.<br />
<br />
So, after a restart, after all systems have restarted, run postboot, which does something like:<br />
<br />
for f in `df -k | grep /dev/vn | awk '{print $6}'` ; do chmod 0666 $f/dev/null ; done<br />
<br />
Later, after the system is up, if anyone ever complains about their /dev/null permissions for any reason ... really if anyone mentions<br />
/dev/null in any way, tell them to run:<br />
<br />
chmod 0666 /dev/null<br />
<br />
We have ultimately fixed this by adding chmods to the quad/safe's<br />
<br />
<br />
== mknod in a jail == <br />
<br />
For a customer who wants to run a chroot name server (named) in a jail.<br />
<br />
<pre><br />
cd /mnt/data1/<user directory>/var/named<br />
mknod -c 1 1 null<br />
mknod -c 2 3 random<br />
</pre><br />
<br />
== Postfix Problems ==<br />
<br />
This hasn't happened in a long time, probably no longer valid with modern OS's<br />
<br />
Postfix is an alternate MTA - people replace sendmail with it in much the same way that many people also replace sendmail with qmail. The problem is that if you install postfix inside of a jail, by default it will not work properly - the user will see error messages like this:<br />
<br />
incoming mail generates errors like this:<br />
<br />
mail_queue_enter: create file incoming/298266.55902: File too large<br />
<br />
and further, sending mail generates errors like this:<br />
<br />
postdrop: warning: mail_queue_enter: create file maildrop/627930.56676: File too large<br />
<br />
This is very easy to solve. In fact, the FreeBSD welcome email that is sent out by `jailmake` now contains this block of text:<br />
<br />
- if you plan on installing postfix, email us first and ask for a necessary patch.<br />
<br />
So, anytime you see a support email that says anything about postfix not working, paste that line from the welcome email into your response to them, and right away, forward the postfix patch to them in an email.<br />
<br />
Here are the instructions:<br />
<br />
Simply copy the patch into /usr/ports/mail/postfix/files, rename the file to ‘patch-file_limit.c’ and then cd to /usr/ports/mail/postfix and run `make install`.<br />
<br />
The postfix patch is the very first email in the support email box, and we never delete or save it because we always want it there to forward to people. The instructions in the email are very clear, and it solves their problem 100% of the time.<br />
<br />
<br />
== Problems with `find` ==<br />
<br />
There was a bug in the FreeBSD vn-filesystem code. The bug causes the system to crash or hang when someone runs the `find` command inside their vn-backed filesystem (and even though the bug may have been fixed, due to the i/o overhead, we still don't like find).<br />
<br />
Now, this does not always happen - however, any jail machine with more than 20 systems on it will crash every night if every system on it runs the daily periodic script out of their crontab, and updates their locate database with the find command.<br />
<br />
(the locate database is a small, FreeBSD specific database that is populated nightly from a cron job, and is then used to provide fast answers to the `locate` command)<br />
<br />
In normal systems (32-40 jails on them), all that has to be done is make sure nobody runs the daily periodic. This means that the file /etc/crontab, instead of looking like this:<br />
<br />
<pre># do daily/weekly/monthly maintenance<br />
1 3 * * * root periodic daily<br />
15 4 * * 6 root periodic weekly<br />
30 5 1 * * root periodic monthly</pre><br />
<br />
Needs to look like this:<br />
<br />
<pre># DO NOT UNCOMMENT THESE - contact support@johncompanies.com for details<br />
#1 3 * * * root periodic daily<br />
#15 4 * * 6 root periodic weekly<br />
#30 5 1 * * root periodic monthly<br />
## DO NOT UNCOMMENT THESE ^^^^^^^^</pre><br />
<br />
and the problem will generally not occur at all. New filesystem images are always altered in this way, so all systems are set like this.<br />
<br />
However, if you have enough systems running, even casual uses of find, outside of the periodic scripts, can cause the system to crash. We are seeing this on jail13, which crashes about once per week.<br />
<br />
There is not yet any resolution to this problem.<br />
<br />
So, if you have a freebsd system crash, see if anyone has uncommented their daily periodic line - you can see them for everyone by running:<br />
<br />
/bin/sh<br />
<br />
and then running:<br />
<br />
<pre>for f in `df -k | grep vn | awk '{print $6}'` ; do echo `cat $f/etc/crontab | grep "periodic daily"` $f ; done | more</pre><br />
<br />
(all one line, one command)<br />
<br />
And then edit those /etc/crontabs and comment them out again.<br />
<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
* if you are getting disk full messages for a BSD customer, it's fairly safe to clear out their /usr/ports/distfiles dir<br />
* 4.x: ps and top an only be run by root in these jails. Done on purpose:<br />
As for `ps` and `top` - non-root users can run them, just not successfully because we have locked the permissions on /dev/mem and /dev/kmem to be root-readable only. That is why non-root users cannot successfully run `ps` and `top`.<br />
* user quotas do not work on freebsd jails - you cannot set up quotas at all, and that's that.<br />
* You cannot inject a process into a 4.x jail, only HUP running processes. 6.x and onward you can with jexec<br />
* jails see base machine’s uptime/load when running top/w<br />
* if someone is unable to get in- cant ping, etc- see if they were blocked by castle (DoS), see if their ip is on the system (post reboot it was lost cause wasn’t in the rc.conf). preboot should catch that.<br />
* in FreeBSD you cant su to root unless you belong to wheel group – so if you remove your acct and setup a new one, we have to add it to wheel group (add to /etc/group)<br />
* Dmesg from underlying sys is seen in customers dmesg on jail<br />
* Popper process going crazy @ 40% for 10min = someone who leaves mail on server<br />
* Don’t force umounts on 4.x jails – it crashes the machine, generally ok on newer machines (running md)<br />
* Good book for admin http://search.barnesandnoble.com/booksearch/isbnInquiry.asp?userid=t824VyRAYz&isbn=0596005164&itm=2<br />
* Self-sign ssl cert http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#selfcert<br />
* conversation with Glenn about semaphores and pgsql<br />
<pre>SDBoody: hey, these are valid amounts/figures/increments right:<br />
kern.ipc.semmni=1280<br />
kern.ipc.semmns=1280<br />
<br />
gr8feen: probably... I always forget exactly what those are, so I usually have to look them up<br />
SDBoody: semaphores<br />
SDBoody: i took the current 1024 and added 256 to them<br />
SDBoody: need more for pgsql<br />
gr8feen: I meant the mni and mns parts...<br />
gr8feen: some of those are not ones you just want to add to... hang on a sec and I'll look them up..<br />
gr8feen: what's semmsl set to?<br />
SDBoody: kern.ipc.semmsl: 1024<br />
<br />
SDBoody: kern.ipc.msgseg: 2048 <br />
kern.ipc.msgssz: 8 <br />
kern.ipc.msgtql: 40 <br />
kern.ipc.msgmnb: 2048 <br />
kern.ipc.msgmni: 40 <br />
kern.ipc.msgmax: 16384 <br />
kern.ipc.semaem: 16384 <br />
kern.ipc.semvmx: 65534 <br />
kern.ipc.semusz: 152 <br />
kern.ipc.semume: 10 <br />
kern.ipc.semopm: 100 <br />
kern.ipc.semmsl: 1024 <br />
kern.ipc.semmnu: 512 <br />
kern.ipc.semmns: 1024 <br />
kern.ipc.semmni: 1024 <br />
kern.ipc.semmap: 768 <br />
kern.ipc.shm_allow_removed: 0 <br />
kern.ipc.shm_use_phys: 1 <br />
kern.ipc.shmall: 262144 <br />
kern.ipc.shmseg: 256 <br />
kern.ipc.shmmni: 784 <br />
kern.ipc.shmmin: 1 <br />
kern.ipc.shmmax: 536870912 <br />
kern.ipc.maxsockets: 25600 <br />
<br />
gr8feen: ok...msl is max per id, mni is max ids, mns is max number of semaphores... so you probably want something like mns = mni * msl<br />
gr8feen: which one did you run out of?<br />
SDBoody: not sure how to tell- ipcs shows the sems in use add up to 1024<br />
SDBoody: there are 59 entries<br />
gr8feen: I'm assuming you tried to start postgres and it failed?<br />
SDBoody: yes<br />
gr8feen: it should have logged why, somewhere..<br />
gr8feen: if I recall, it'll tell you which one it ran out of<br />
SDBoody: > DETAIL: Failed system call was semget(1, 17, 03600).<br />
<br />
gr8feen: so it wanted an id with 17 semaphores...I'd start by making mns = 17*mni and leave mni and mnl set to what they are now and see what it does<br />
SDBoody: i think mni is plenty high<br />
SDBoody: ok, more reasonable 17408<br />
gr8feen: yeah...just change that one and see how it goes..<br />
SDBoody: and leave mni alone at 1024?<br />
gr8feen: yeah...mni id the max number of ids...but if it's trying to get something like 17 per id, your going to hit mns before you hit anything else<br />
SDBoody: right, but doesn't hurt to have it that high (assuming)<br />
<br />
gr8feen: not really... I think those get allocated out of ram that you cant page out, but it's still such a small amount that it really doesn't matter<br />
SDBoody: looks like that worked, thx!<br />
gr8feen: cool<br />
<br />
see semaphores: ipcs -a -s</pre><br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
== Making new customer VE (vm) ==<br />
<br />
This applies only to new virts >= 4.x<br />
<br />
grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.<br />
<br />
1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
also, ping the IP address.<br />
<br />
2. put CT on whichever partition has more space<br />
df -h<br />
<br />
3. vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> ) <br />
vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@johncompanies.com centos-6-x86_64 LM<br />
<br />
4. copy veid, dir, ip and password to pending customer screen. activate customer<br />
<br />
== Making new customer VE (vemakexxx) ==<br />
<br />
This applies to older virts with old templates. This should probably not be used at all anymore.<br />
<br />
<br />
1. look thru hist for ip<br />
<br />
2. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
<br />
3. put ve on whichever partition has more space<br />
vemakerh9 <veid> <ip> <hostname> <mount> <email> [gb disk]; <256|384|512> <veid><br />
vemakerh9 866 69.55.226.109 ngentu.com /vz1 ayo@ngantu.com,asd@asd.com 5; 256 866<br />
<br />
4. copy (veid), dir, and ip to pending customer screen (pass set to p455agfa)<br />
<br />
<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrateonline/migrate failures (migrate manually) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
You may also see migration failing due to quota issues.<br />
<br />
You can try to resolve by copying any quota file into the file you need:<br />
<br />
cp /var/vzquota/quota.1 /var/vzquota/quota.xxx<br />
<br />
If it complains about quota running you should then be able to stop it<br />
<br />
vzquota off xxxx<br />
<br />
If all else fails, migrate to a new VEID<br />
i.e. 1234 becomes 12341<br />
<br />
<br />
If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
'''Note:''' don't forget to fix /usr/local/sbin/backup.config for new disk location<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
=== Cant change file /etc/sysconfig/network ===<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
=== What template is user using ===<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
=== Edits needed to run java ===<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
=== Set time zone ===<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
=== Failing shm_open calls ===<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
=== Debian can't get on the network ===<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
=== CentOS can't get on the network ===<br />
<br />
When CentOS system was running, but can't get on the network, they probably updated iproute via yum.<br />
<br />
Copy an older version of the iproute rpm to their VPS.<br />
<br />
<pre>cp -p /root/iproute/iproute-2.6.32-23.el6.<i686>.rpm /vz2/private/<1527>/fs/root</pre><br />
<br />
Enter their ve, install rpm, restart network, and prevent yum from updating iproute.<br />
<br />
<pre><br />
ve <1527><br />
rpm -i --force iproute-2.6.32-23.el6.<i686>.rpm<br />
rpm -q iproute<br />
rpm -e <new iproute rpm><br />
service network restart<br />
echo "exclude=iproute" >>/etc/yum/ip-exclude<br />
echo "include=/etc/yum/ip-exclude" >>/etc/yum.conf<br />
</pre><br />
<br />
<br />
<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
----<br />
<br />
* uptime/load that vz users see isn’t what we see- it's their own load<br />
* Vz overwrites resolve.conf on every boot to what’s in ve config file. Also overwrites hostname<br />
* Quotas in vz: set Gid_limit in vz config file and restart ve. Errors leading up to this will be out of files but no inode or space issues. set to 2000 in conf, and restart vz<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
To create a new Linux VPS<br />
<br />
* get an IP address from mgmt screen<br />
* run df -h to figure out which disk has enough space<br />
* run vm with no arguments to get syntax and exact template name<br />
* use col0xxxx for hostname if they don’t give you a hostname<br />
* copy over dir, veid, ip and password to pending customer screen<br />
<br />
vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> )<br />
<br />
== disable ==<br />
To disable a customers Virtuozzo VPS server.<br />
<br />
* stop the server<br />
vzctl stop <veid><br />
* prevent restart on server reboot and user restart via Control Panel<br />
vzctl set <veid> --disabled=yes --offline_management=no --save<br />
<br />
== cancelve ==<br />
When a customer cancels all service.<br />
<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
<br />
display the contents of the vz config file for the <veid><br />
<br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
<br />
List all virtual environments running on this machine.<br />
<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
<br />
show processes running under this <veid>.<br />
<br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
<br />
Restart a <veid>.<br />
<br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
<br />
enter a <veid> as root.<br />
<br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
<br />
execute a single command in the <veid> environment.<br />
<br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem VEID <256|512|768|1024|1280|1536|2048> [burst GB]<br />
adjusts the memory resources for the VE. If this is on VZ ver >4, you will also be prompted to provide a burst ram amount. Sample usage:<br />
<br />
<pre># setmem 21671 1024 4<br />
/usr/sbin/vzctl set 21671 --ram 1024m --swap 3072m --save<br />
</pre><br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>75.80.128.147https://wiki.jcihosting.com/index.php?title=VPS_Management&diff=2223VPS Management2020-04-03T17:25:02Z<p>75.80.128.147: /* Ganeti Insall a VPS */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
== Moving a server to a different account in the management system ==<br />
<br />
This is a tough one. You gotta touch a bunch of tables in the mysql database.<br />
<pre><br />
systems<br />
subs<br />
ipmap<br />
</pre><br />
<br />
basically, get the sysid and go off that:<br />
<br />
<pre><br />
update systems set cid='newcid' where sysid=xxxxx<br />
update subs set cid='newcid' where sysid=xxxxx<br />
update ipmap set cid='newcid' where sysid=xxxxx<br />
</pre><br />
<br />
You then need to rename the server disk space to change the old ID to the new one.<br />
<br />
For a jail -<br />
<pre><br />
stopjail <hostname><br />
mv 69.55.2XX.XXX-col0<oldid> 69.55.2XX.XXX-col0<newid><br />
mv 69.55.2XX.XXX-col0<oldid>-DIR 69.55.2XX.XXX-col0<newid>-DIR<br />
vi /usr/local/jail/bin/quad1 [Update the ID in the quad file]<br />
startjail <hostname><br />
</pre><br />
<br />
For a virt -<br />
You will have to <br />
* stop the virt <br />
* rename the directories <br />
* modify the config file /vzconf/<veid>.conf <br />
* rename the virt conf file <br />
* restart the new virt.<br />
<br />
that ought to do it..<br />
<br />
<br />
== Re-enable a customer account in the management system ==<br />
<br />
For example to re-instate col03438, In mysql:<br />
<br />
<pre><br />
show tables ;<br />
select * from customers where cid='col03438' ;<br />
update customers set status=1 where cid='col03438' ;<br />
select * from systems where cid='col03438' ;<br />
update systems set stop_date=NULL where cid='col03438' ;<br />
update systems set cancel_date=NULL where cid='col03438' ;<br />
update systems set status=1 where cid='col03438' ;<br />
select * from subs where cid='col03438' ;<br />
update subs set stop_date=NULL where cid='col03438' ;<br />
update subs set status=1 where cid='col03438' <br />
<br />
</pre><br />
<br />
= Ganeti VPS =<br />
<br />
This is how to create a VPS on the Lamphost ganeti cluster.<br />
I have created scripts for creating a FreeBSD VPS and one for<br />
creating a Linux VPS.<br />
<br />
== Ganeti Install a VPS ==<br />
<br />
To create a linux VPS use<br />
<pre><br />
/root/new <name> <disk size> <memory> <IP Addr> [<OS>]<br />
<br />
( for example /root/new vps.example.com 30 2 69.55.238.10 centos-8 )<br />
</pre><br />
<br />
To create a FreeBSD VPS, we have to do a full install. This will create a <br />
new VPS.<br />
<br />
<pre><br />
/root/newfreebsd12 <name> <disk size> <memory> <IP Addr> [<OS>]<br />
</pre><br />
<br />
You will then have to do the install via VNC.<br />
<br />
You need to customize the install by adding a "jci" account and a customer<br />
account (col0XXXX) as well. <br />
<br />
<br> <br />
You will need to install sudo.<br />
<pre> <br />
pkg install sudo # install sudo package<br />
visudo # to add wheel account to sudoers<br />
vi /etc/group # add jci and user acct to wheel<br />
</pre><br />
<br />
<br><br />
You will also have to set up a console in /etc/ttys.<br />
<pre><br />
vi /etc/ttys<br />
</pre><br />
<br />
== Ganeti Re-Insall a VPS ==<br />
<br />
Re-installing the OS will destroy all data on the VPS.<br />
Login to the Ganeti master node (currently gn1).<br />
<br />
To reinstall the OS on a VPS the VPS must be shutdown.<br />
<pre><br />
gnt-instance shutdown <VPSname><br />
</pre><br />
<br />
Do the reinstall.<br />
<pre><br />
gnt-instance reinstall <VPSname><br />
</pre><br />
<br />
Restart the VPS.<br />
<pre><br />
gnt-instance startup <VPSname><br />
</pre><br />
<br />
Now you need to do the customization for the VPS. This is<br />
basically the same as the Dedicated server customization.<br />
<br />
= FreeBSD 10.1 VPS (jail5 and jail6) =<br />
<br />
This is information on how to manage the FreeBSD 10.1 VPS servers on Jail5.<br />
<br />
<br />
== Making new customer VPS ==<br />
<pre><br />
virtmake IP[,IP] <CID> hostname ipfw# email {FSP|FB|FS|FM|FP} [size in GB]<br />
</pre><br />
<br />
Note: <br />
For now, CID must be unique in the last 2 digits of the CID.<br />
<br />
Must do a "vm start <CID>" after creating a VPS.<br />
<br />
== Starting VPS ==<br />
To start all vms on the system<br />
<pre><br />
vm start<br />
</pre><br />
<br />
To start a single VPS <br />
<pre><br />
vm start <CID><br />
</pre><br />
<br />
== List running VPS ==<br />
<pre><br />
vm status<br />
</pre><br />
<br />
== Attach to a user VPS console ==<br />
<pre><br />
vm attach <CID><br />
</pre><br />
<br />
<br />
To detach from a console, type tilde control-d (~ ctl-d).<br />
<br />
== Stop a VPS ==<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
== Restart a VPS ==<br />
<pre><br />
vm restart <CID><br />
</pre><br />
<br />
<br />
== Modify a VPS ==<br />
<br />
To modify a VPS config, stop the VPS<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
Mount the VPS data file<br />
<pre><br />
vm mount <CID><br />
<br />
cd /data/vm/private/<CID>/mount<br />
</pre><br />
<br />
Modify the user files necessary.<br />
<br />
Restart the VPS.<br />
<br />
<pre><br />
vm umount <CID><br />
<br />
vm start <CID><br />
</pre><br />
<br />
== Notes on a new FBSD 10.x VPS (jail6) ==<br />
<br />
<pre><br />
#! /bin/sh<br />
#<br />
# new <colo#> <disk><br />
<br />
<br />
memory=1024<br />
<br />
x="$1"<br />
<br />
disk="$2"<br />
if [ "y" == "y$disk" ] ; then<br />
disk=60<br />
fi<br />
<br />
disk2=`expr $disk - 4`<br />
<br />
<br />
vm image provision 04dcdf96-d843-11e6-b131-001e67fcc64f col0$x<br />
# vm configure col0$x<br />
sed -e "s/memory=256M/memory=${memory}M/" /vms/col01114/col01114.conf >/vms/col0$x/col0$x.conf<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
# vm start col0$x<br />
<br />
<br />
exit<br />
<br />
vm console col0$x<br />
<br />
#<br />
# to increase the customer disk<br />
#<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
<br />
## on the customer VPS<br />
<br />
gpart recover vtbd0<br />
gpart show vtbd0<br />
swapoff /dev/vtbd0p3<br />
gpart delete -i 3 vtbd0<br />
gpart show vtbd0<br />
sysctl kern.geom.debugflags=16<br />
gpart resize -i 2 -a 4k -s $disk2\G vtbd0<br />
<br />
growfs /dev/vtbd0p2<br />
<br />
gpart show vtbd0<br />
gpart add -t freebsd-swap -a 4k vtbd0<br />
gpart show vtbd0<br />
swapon /dev/vtbd0p3<br />
<br />
df -h<br />
reboot<br />
<br />
vi /etc/rc.conf<br />
hostname<br />
IP Addr<br />
passwd root<br />
passwd user<br />
clear history<br />
cp /dev/null /var/log/utx.log<br />
cp /dev/null ~root/.history<br />
history -c<br />
<br />
</pre><br />
<br />
= FreeBSD Jails =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Making new customer jail 4.x ==<br />
<br />
1. run [[#js|js]] to figure out which partition, IP to put it on, find an unused vn, and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vnX hostname shorthost quad/safe# ipfw# email [gb disk, default 4]</tt><br />
<br />
jail14# /tmp/jailmake 69.55.226.152 /mnt/data2/69.55.226.152-col00182 vn23 negev.cerastes.org negev 4 182 cerastes@cerastes.org<br />
<br />
<br />
== Making new customer jail 6.x ==<br />
<br />
1. run [[#js|js]] to figure out which gvinum vols are free (and which mnt (disk) each gvinum goes with/to), IP to put it on and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vN[,vN] hostname shorthost quad/safe# ipfw# email</tt><br />
<br />
jail19# jailmake 69.55.236.17 /mnt/data1/69.55.236.17-col01574 v10 uncle-leo.commadev.com uncle-leo 1 1574 lee@commadev.com,lee@gmail.com<br />
<br />
<br />
== Changing an ip for freebsd VPS ==<br />
<br />
*stopjail (hostname)<br />
*on systems using vnfile or mdfile: rename vnfile/mdfile, using new ip<br />
*rename dir using new ip<br />
*Edit quad (make sure to update all lines: directory, vn/mdfile)<br />
*buildsafe (not necessary on systems having only quad1)<br />
*if you're adding an ip not already on the host machine: <tt>ipadd x.x.x.x</tt><br />
*startjail (hostname)<br />
if backups: <tt>mvbackups</tt> ...<br />
*Edit dir/ip in mgmt<br />
*if any rules, update any firewall rules to use new ip (careful not to make sure there arent multiple rules per ipfw#- search by colo#)<br />
*(if customer asks and has domains) update their domains on ns1c<br />
*optional: update ipfw counters on the host server<br />
<br />
<br />
== Rename a gconcat vol ==<br />
<br />
*stop jail, unmount everything<br />
*gconcat stop vol<br />
*gconcat clear /dev/gvinum/vN<br />
*gconcat clear /dev/gvinum/vN+1<br />
*gconcat label -v newname /dev/gvinum/vN /dev/gvinum/vN+1<br />
*bsdlabel /dev/concat/newname<br />
(make sure a partition is not fstype unused, if so change to: 4.2BSD)<br />
<br />
<br />
== Remaking a VPS (on same jail) ==<br />
<br />
# take him down (stopjail)<br />
# make note of host, vn/gvin/md, ip<br />
# remove from quad/safe<br />
# move vnfile (rename)<br />
# remove dir (jailmake will make a new one)<br />
# remove ipfw counts<br />
# jailmake<br />
# update backup script<br />
# update db with new dir/ip (use the “move” link, mark old as stopped today)<br />
# update firewall if changed ip<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
<br />
== Adding disk to a 4.x jail ==<br />
<br />
NOTE: if you unmount before vnconfig, re-vnconfig then unmount then vnunconfig<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them.<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to dump to a different partition and create the new vn on the current partition. If no space to make make dump, then move old vn’s to backup or worse, dump to backup.<br />
<br />
<br />
2. <tt>g <customerID></tt> to get the info<br />
<br />
<br />
3. make vnfile, newfs and mount. USE CAUTION not to overwrite existing vn. <br><br />
Ex: <tt>vnconfig –T -S 9g -s labels -c /dev/vn22 /mnt/data2/65.214.160.117-col00496<br><br />
disklabel -r -w vn30 auto<br><br />
newfs /dev/vn30c<br><br />
mkdir /mnt/data2/65.214.160.117-col00496-DIR<br><br />
mount /dev/vn30c /mnt/data2/69.55.237.26-col00241-DIR<br><br />
cd /mnt/data2/65.214.160.117-col00496-DIR</tt><br />
<br />
<br />
4. <tt>jailkill <hostname></tt><br />
<br />
<br />
5. <tt>dump -0a -f - /dev/vn1 | restore -r -f -</tt> <br />
<br />
<br />
6. <tt>rm restoresymtable</tt><br />
<br />
<br />
7. unmount and unvnconfig old system:<br><br />
<tt>umount /dev/vnNNc <br><br />
vnconfig –u /dev/vnNN</tt><br />
<br />
<br />
8. move vnfile. Ex: <tt>mv /mnt/data1/69.55.237.26-col00241 /mnt/data1/old-col00241-vnfile-20110312-noarchive</tt><br />
<br />
<br />
9. remove old dir: <tt>rmdir /mnt/data1/69.55.237.26-col00241-DIR</tt><br />
<br />
<br />
10. edit quad/safe to point to new location, run <tt>[[#buildsafe|buildsafe]]</tt><br />
<br />
<br />
11. start jail: <tt>[[#startjail|startjail]] <hostname></tt><br />
<br />
<br />
12. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
13. mv backups if necessary<br />
<br />
WARNING: if you restore to unmounted vn then you’re actually restoring to /mnt/data1<br />
you can tar and do above to recover<br />
<br />
NOTE: you can’t move a customer to a system that has a base sys that is diff from the one they came from (BSD 4.8 to 4.5 = won’t work)<br />
<br />
NOTE: we don’t mount procfs's to anyone’s system by default.<br />
<br />
NOTE: mount_nulls: <br />
also goofy mount_nulls seen in df -k on jail2<br><br />
also needs to be done in safe and quad<br><br />
if he wants another, use safe/quad as example and add it to file. also exec command at prompt to add mount (no reboot, remount or rejail necessary)<br />
<br />
== Moving customer to another jail machine ==<br />
<br />
Systems should only be moved between similarly-versioned jails. If you cannot get a match, you should go to freebsd.org and see what the differences were between the version you're moving from and the new version. Many of the updates will be to drivers and kernels - these don't affect the customer. Perhaps they updated traceroute or a library...you should copy the new/changed files into the VPS/jail on the new host. An alternate method for moving to a different version would be to buildworld to bring their full distribution up to date. Optional steps for this kind of upgrade are included below.<br />
<br />
1. <tt>g <customerID></tt><br />
<br />
<br />
2. <tt>[[#jailkill|jailkill]] <hostname></tt><br />
<br />
<br />
3. create new device on target system<br><br />
4.x:<br><br />
<pre>vnconfig –T –S 4g –s labels –c /dev/vn1 /mnt/data1/69.55.22x.x-col00XXX<br />
disklabel –r –w vn1 auto<br />
newfs /dev/vn1c</pre><br />
<br />
6.x:<br><br />
<pre>bsdlabel -r -w /dev/gvinum/v1<br />
newfs /dev/gvinum/v1a<br />
or <br />
gconcat label v1-v3 /dev/gvinum/v1 /dev/gvinum/v2 /dev/gvinum/v3<br />
bsdlabel -r -w /dev/concat/v1-v3<br />
newfs /dev/concat/v1-v3a</pre><br />
<br />
7.x+:<br><br />
Run jailmakeempty then skip step 4 & 7 below<br />
<br />
<br />
4. make and mount the dir on new system. <br><br />
Ex: <pre>mkdir /mnt/data2/69.55.230.3-col00123-DIR<br />
chmod 755 /mnt/data2/69.55.230.3-col00123-DIR<br />
mount [device] /mnt/data2/69.55.230.3-col00123-DIR</pre><br />
<br />
<br />
5. [[#stopjail|stopjail]] <hostname> 1<br />
<br />
<br />
6. dump fs to new system. ex: <tt>[[#dumpremoterestore|dumpremoterestore]] /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR</tt> (make sure you can ssh as root on remote machine) <br />
<br />
<br />
6a. OPTIONAL BUILDWORLD:<br><br />
cd /usr/src<br />
make world DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
cd etc<br />
make distribution DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
(you may have to rm an openssh file and re-make dist)<br />
<br />
rm -rf /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/periodic/daily/400.status-disks<br />
vi /etc/periodic/security/100.chksetuid<br />
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt><br><br />
with: <tt>MP='/' (use single quotes)</tt><br />
<br />
Ask user if they want ports overwritten with current, if yes:<br />
cp -r /usr/ports /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/usr<br />
<br />
on source: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d .<br />
(copy in all info)<br />
<br />
on target: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d . <br />
(paste all info)<br />
<br />
on source: <br />
cat /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group<br />
(copy in all info)<br />
<br />
on target:<br />
cat > /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group <br />
(paste all info)<br />
<br />
<br />
7. edit quad on source system copy over entries for the jail to the target system <br />
take care that the vn/gvinum/gconcat/md devices on the target system are’t in use and that the /mnt/dataN path doesn’t need changing<br />
<br />
<br />
8. run [[#buildsafe|buildsafe]] (if this is <=6.x) on target system (copies changes made to quad into safe file)<br />
<br />
<br />
9. remove ip from source system. Ex: <tt>ipdel 69.55.230.3</tt><br />
<br />
<br />
10. add ip to target system. Ex: <tt>ipadd 69.55.230.3</tt><br />
<br />
<br />
11. start new system, with [[#startjail|startjail]] or manually, pasting entries found from running <tt>g <customerID></tt> on the new system<br />
<br />
<br />
12. run <tt>[[#canceljail|canceljail]] col0xxxx</tt> on source system. This should prompt you to remove backups if any existed. Do not let it do this, rather:<br />
<br />
<br />
13. if backups existed move them to the new host via the <tt>[[#mvbackups_.28freebsd.29|mvbackups]] script</tt><br />
<br />
<br />
14. edit quad on source system, edit comment to reflect a move rather than a cancel, ex: <tt># moved to jail2 col00241</tt><br />
<br />
<br />
15. edit mgmt to reflect new host and dir for new system<br />
<br />
<br />
16. optional add ipfw rules on new system. Ex:<br />
<tt>ipfw add 01231 count ip from 69.55.230.2 to any<br />
ipfw add 01232 count ip from any to 69.55.230.2</tt><br />
<br />
== Increasing inodes for a VPS ==<br />
<br />
when doing the newfs: <br />
newfs -i 4096 ...<br />
<br />
<br />
== Jail services unable to contact services within same jail ==<br />
<br />
This is due to a messed up routing table (only seen it on jail2 before). <br />
<br />
route delete 69.55.228.65/32 route add -net 69.55.228.65/32 -iface fxp0 -nostatic -cloning route add 69.55.228.65 -iface lo0<br />
<br />
<br />
== /dev/null permission resets == <br />
<br />
Applies to 4.x jails only<br />
<br />
For some reason that I do not understand at all, the /dev/null node in customer jails, after the system is restarted, often reverts permissions to 0600 ... which is bad, because non-root processes cannot redirect to /dev/null anymore. In fact, a fair number of server daemons will complain and/or fail if this is the case.<br />
<br />
So, after a restart, after all systems have restarted, run postboot, which does something like:<br />
<br />
for f in `df -k | grep /dev/vn | awk '{print $6}'` ; do chmod 0666 $f/dev/null ; done<br />
<br />
Later, after the system is up, if anyone ever complains about their /dev/null permissions for any reason ... really if anyone mentions<br />
/dev/null in any way, tell them to run:<br />
<br />
chmod 0666 /dev/null<br />
<br />
We have ultimately fixed this by adding chmods to the quad/safe's<br />
<br />
<br />
== mknod in a jail == <br />
<br />
For a customer who wants to run a chroot name server (named) in a jail.<br />
<br />
<pre><br />
cd /mnt/data1/<user directory>/var/named<br />
mknod -c 1 1 null<br />
mknod -c 2 3 random<br />
</pre><br />
<br />
== Postfix Problems ==<br />
<br />
This hasn't happened in a long time, probably no longer valid with modern OS's<br />
<br />
Postfix is an alternate MTA - people replace sendmail with it in much the same way that many people also replace sendmail with qmail. The problem is that if you install postfix inside of a jail, by default it will not work properly - the user will see error messages like this:<br />
<br />
incoming mail generates errors like this:<br />
<br />
mail_queue_enter: create file incoming/298266.55902: File too large<br />
<br />
and further, sending mail generates errors like this:<br />
<br />
postdrop: warning: mail_queue_enter: create file maildrop/627930.56676: File too large<br />
<br />
This is very easy to solve. In fact, the FreeBSD welcome email that is sent out by `jailmake` now contains this block of text:<br />
<br />
- if you plan on installing postfix, email us first and ask for a necessary patch.<br />
<br />
So, anytime you see a support email that says anything about postfix not working, paste that line from the welcome email into your response to them, and right away, forward the postfix patch to them in an email.<br />
<br />
Here are the instructions:<br />
<br />
Simply copy the patch into /usr/ports/mail/postfix/files, rename the file to ‘patch-file_limit.c’ and then cd to /usr/ports/mail/postfix and run `make install`.<br />
<br />
The postfix patch is the very first email in the support email box, and we never delete or save it because we always want it there to forward to people. The instructions in the email are very clear, and it solves their problem 100% of the time.<br />
<br />
<br />
== Problems with `find` ==<br />
<br />
There was a bug in the FreeBSD vn-filesystem code. The bug causes the system to crash or hang when someone runs the `find` command inside their vn-backed filesystem (and even though the bug may have been fixed, due to the i/o overhead, we still don't like find).<br />
<br />
Now, this does not always happen - however, any jail machine with more than 20 systems on it will crash every night if every system on it runs the daily periodic script out of their crontab, and updates their locate database with the find command.<br />
<br />
(the locate database is a small, FreeBSD specific database that is populated nightly from a cron job, and is then used to provide fast answers to the `locate` command)<br />
<br />
In normal systems (32-40 jails on them), all that has to be done is make sure nobody runs the daily periodic. This means that the file /etc/crontab, instead of looking like this:<br />
<br />
<pre># do daily/weekly/monthly maintenance<br />
1 3 * * * root periodic daily<br />
15 4 * * 6 root periodic weekly<br />
30 5 1 * * root periodic monthly</pre><br />
<br />
Needs to look like this:<br />
<br />
<pre># DO NOT UNCOMMENT THESE - contact support@johncompanies.com for details<br />
#1 3 * * * root periodic daily<br />
#15 4 * * 6 root periodic weekly<br />
#30 5 1 * * root periodic monthly<br />
## DO NOT UNCOMMENT THESE ^^^^^^^^</pre><br />
<br />
and the problem will generally not occur at all. New filesystem images are always altered in this way, so all systems are set like this.<br />
<br />
However, if you have enough systems running, even casual uses of find, outside of the periodic scripts, can cause the system to crash. We are seeing this on jail13, which crashes about once per week.<br />
<br />
There is not yet any resolution to this problem.<br />
<br />
So, if you have a freebsd system crash, see if anyone has uncommented their daily periodic line - you can see them for everyone by running:<br />
<br />
/bin/sh<br />
<br />
and then running:<br />
<br />
<pre>for f in `df -k | grep vn | awk '{print $6}'` ; do echo `cat $f/etc/crontab | grep "periodic daily"` $f ; done | more</pre><br />
<br />
(all one line, one command)<br />
<br />
And then edit those /etc/crontabs and comment them out again.<br />
<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
* if you are getting disk full messages for a BSD customer, it's fairly safe to clear out their /usr/ports/distfiles dir<br />
* 4.x: ps and top an only be run by root in these jails. Done on purpose:<br />
As for `ps` and `top` - non-root users can run them, just not successfully because we have locked the permissions on /dev/mem and /dev/kmem to be root-readable only. That is why non-root users cannot successfully run `ps` and `top`.<br />
* user quotas do not work on freebsd jails - you cannot set up quotas at all, and that's that.<br />
* You cannot inject a process into a 4.x jail, only HUP running processes. 6.x and onward you can with jexec<br />
* jails see base machine’s uptime/load when running top/w<br />
* if someone is unable to get in- cant ping, etc- see if they were blocked by castle (DoS), see if their ip is on the system (post reboot it was lost cause wasn’t in the rc.conf). preboot should catch that.<br />
* in FreeBSD you cant su to root unless you belong to wheel group – so if you remove your acct and setup a new one, we have to add it to wheel group (add to /etc/group)<br />
* Dmesg from underlying sys is seen in customers dmesg on jail<br />
* Popper process going crazy @ 40% for 10min = someone who leaves mail on server<br />
* Don’t force umounts on 4.x jails – it crashes the machine, generally ok on newer machines (running md)<br />
* Good book for admin http://search.barnesandnoble.com/booksearch/isbnInquiry.asp?userid=t824VyRAYz&isbn=0596005164&itm=2<br />
* Self-sign ssl cert http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#selfcert<br />
* conversation with Glenn about semaphores and pgsql<br />
<pre>SDBoody: hey, these are valid amounts/figures/increments right:<br />
kern.ipc.semmni=1280<br />
kern.ipc.semmns=1280<br />
<br />
gr8feen: probably... I always forget exactly what those are, so I usually have to look them up<br />
SDBoody: semaphores<br />
SDBoody: i took the current 1024 and added 256 to them<br />
SDBoody: need more for pgsql<br />
gr8feen: I meant the mni and mns parts...<br />
gr8feen: some of those are not ones you just want to add to... hang on a sec and I'll look them up..<br />
gr8feen: what's semmsl set to?<br />
SDBoody: kern.ipc.semmsl: 1024<br />
<br />
SDBoody: kern.ipc.msgseg: 2048 <br />
kern.ipc.msgssz: 8 <br />
kern.ipc.msgtql: 40 <br />
kern.ipc.msgmnb: 2048 <br />
kern.ipc.msgmni: 40 <br />
kern.ipc.msgmax: 16384 <br />
kern.ipc.semaem: 16384 <br />
kern.ipc.semvmx: 65534 <br />
kern.ipc.semusz: 152 <br />
kern.ipc.semume: 10 <br />
kern.ipc.semopm: 100 <br />
kern.ipc.semmsl: 1024 <br />
kern.ipc.semmnu: 512 <br />
kern.ipc.semmns: 1024 <br />
kern.ipc.semmni: 1024 <br />
kern.ipc.semmap: 768 <br />
kern.ipc.shm_allow_removed: 0 <br />
kern.ipc.shm_use_phys: 1 <br />
kern.ipc.shmall: 262144 <br />
kern.ipc.shmseg: 256 <br />
kern.ipc.shmmni: 784 <br />
kern.ipc.shmmin: 1 <br />
kern.ipc.shmmax: 536870912 <br />
kern.ipc.maxsockets: 25600 <br />
<br />
gr8feen: ok...msl is max per id, mni is max ids, mns is max number of semaphores... so you probably want something like mns = mni * msl<br />
gr8feen: which one did you run out of?<br />
SDBoody: not sure how to tell- ipcs shows the sems in use add up to 1024<br />
SDBoody: there are 59 entries<br />
gr8feen: I'm assuming you tried to start postgres and it failed?<br />
SDBoody: yes<br />
gr8feen: it should have logged why, somewhere..<br />
gr8feen: if I recall, it'll tell you which one it ran out of<br />
SDBoody: > DETAIL: Failed system call was semget(1, 17, 03600).<br />
<br />
gr8feen: so it wanted an id with 17 semaphores...I'd start by making mns = 17*mni and leave mni and mnl set to what they are now and see what it does<br />
SDBoody: i think mni is plenty high<br />
SDBoody: ok, more reasonable 17408<br />
gr8feen: yeah...just change that one and see how it goes..<br />
SDBoody: and leave mni alone at 1024?<br />
gr8feen: yeah...mni id the max number of ids...but if it's trying to get something like 17 per id, your going to hit mns before you hit anything else<br />
SDBoody: right, but doesn't hurt to have it that high (assuming)<br />
<br />
gr8feen: not really... I think those get allocated out of ram that you cant page out, but it's still such a small amount that it really doesn't matter<br />
SDBoody: looks like that worked, thx!<br />
gr8feen: cool<br />
<br />
see semaphores: ipcs -a -s</pre><br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
== Making new customer VE (vm) ==<br />
<br />
This applies only to new virts >= 4.x<br />
<br />
grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.<br />
<br />
1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
also, ping the IP address.<br />
<br />
2. put CT on whichever partition has more space<br />
df -h<br />
<br />
3. vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> ) <br />
vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@johncompanies.com centos-6-x86_64 LM<br />
<br />
4. copy veid, dir, ip and password to pending customer screen. activate customer<br />
<br />
== Making new customer VE (vemakexxx) ==<br />
<br />
This applies to older virts with old templates. This should probably not be used at all anymore.<br />
<br />
<br />
1. look thru hist for ip<br />
<br />
2. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
<br />
3. put ve on whichever partition has more space<br />
vemakerh9 <veid> <ip> <hostname> <mount> <email> [gb disk]; <256|384|512> <veid><br />
vemakerh9 866 69.55.226.109 ngentu.com /vz1 ayo@ngantu.com,asd@asd.com 5; 256 866<br />
<br />
4. copy (veid), dir, and ip to pending customer screen (pass set to p455agfa)<br />
<br />
<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrateonline/migrate failures (migrate manually) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
You may also see migration failing due to quota issues.<br />
<br />
You can try to resolve by copying any quota file into the file you need:<br />
<br />
cp /var/vzquota/quota.1 /var/vzquota/quota.xxx<br />
<br />
If it complains about quota running you should then be able to stop it<br />
<br />
vzquota off xxxx<br />
<br />
If all else fails, migrate to a new VEID<br />
i.e. 1234 becomes 12341<br />
<br />
<br />
If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
'''Note:''' don't forget to fix /usr/local/sbin/backup.config for new disk location<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
=== Cant change file /etc/sysconfig/network ===<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
=== What template is user using ===<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
=== Edits needed to run java ===<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
=== Set time zone ===<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
=== Failing shm_open calls ===<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
=== Debian can't get on the network ===<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
=== CentOS can't get on the network ===<br />
<br />
When CentOS system was running, but can't get on the network, they probably updated iproute via yum.<br />
<br />
Copy an older version of the iproute rpm to their VPS.<br />
<br />
<pre>cp -p /root/iproute/iproute-2.6.32-23.el6.<i686>.rpm /vz2/private/<1527>/fs/root</pre><br />
<br />
Enter their ve, install rpm, restart network, and prevent yum from updating iproute.<br />
<br />
<pre><br />
ve <1527><br />
rpm -i --force iproute-2.6.32-23.el6.<i686>.rpm<br />
rpm -q iproute<br />
rpm -e <new iproute rpm><br />
service network restart<br />
echo "exclude=iproute" >>/etc/yum/ip-exclude<br />
echo "include=/etc/yum/ip-exclude" >>/etc/yum.conf<br />
</pre><br />
<br />
<br />
<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
----<br />
<br />
* uptime/load that vz users see isn’t what we see- it's their own load<br />
* Vz overwrites resolve.conf on every boot to what’s in ve config file. Also overwrites hostname<br />
* Quotas in vz: set Gid_limit in vz config file and restart ve. Errors leading up to this will be out of files but no inode or space issues. set to 2000 in conf, and restart vz<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
To create a new Linux VPS<br />
<br />
* get an IP address from mgmt screen<br />
* run df -h to figure out which disk has enough space<br />
* run vm with no arguments to get syntax and exact template name<br />
* use col0xxxx for hostname if they don’t give you a hostname<br />
* copy over dir, veid, ip and password to pending customer screen<br />
<br />
vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> )<br />
<br />
== disable ==<br />
To disable a customers Virtuozzo VPS server.<br />
<br />
* stop the server<br />
vzctl stop <veid><br />
* prevent restart on server reboot and user restart via Control Panel<br />
vzctl set <veid> --disabled=yes --offline_management=no --save<br />
<br />
== cancelve ==<br />
When a customer cancels all service.<br />
<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
<br />
display the contents of the vz config file for the <veid><br />
<br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
<br />
List all virtual environments running on this machine.<br />
<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
<br />
show processes running under this <veid>.<br />
<br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
<br />
Restart a <veid>.<br />
<br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
<br />
enter a <veid> as root.<br />
<br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
<br />
execute a single command in the <veid> environment.<br />
<br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem VEID <256|512|768|1024|1280|1536|2048> [burst GB]<br />
adjusts the memory resources for the VE. If this is on VZ ver >4, you will also be prompted to provide a burst ram amount. Sample usage:<br />
<br />
<pre># setmem 21671 1024 4<br />
/usr/sbin/vzctl set 21671 --ram 1024m --swap 3072m --save<br />
</pre><br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>75.80.128.147https://wiki.jcihosting.com/index.php?title=VPS_Management&diff=2222VPS Management2020-04-03T17:22:57Z<p>75.80.128.147: /* Ganeti VPS */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
== Moving a server to a different account in the management system ==<br />
<br />
This is a tough one. You gotta touch a bunch of tables in the mysql database.<br />
<pre><br />
systems<br />
subs<br />
ipmap<br />
</pre><br />
<br />
basically, get the sysid and go off that:<br />
<br />
<pre><br />
update systems set cid='newcid' where sysid=xxxxx<br />
update subs set cid='newcid' where sysid=xxxxx<br />
update ipmap set cid='newcid' where sysid=xxxxx<br />
</pre><br />
<br />
You then need to rename the server disk space to change the old ID to the new one.<br />
<br />
For a jail -<br />
<pre><br />
stopjail <hostname><br />
mv 69.55.2XX.XXX-col0<oldid> 69.55.2XX.XXX-col0<newid><br />
mv 69.55.2XX.XXX-col0<oldid>-DIR 69.55.2XX.XXX-col0<newid>-DIR<br />
vi /usr/local/jail/bin/quad1 [Update the ID in the quad file]<br />
startjail <hostname><br />
</pre><br />
<br />
For a virt -<br />
You will have to <br />
* stop the virt <br />
* rename the directories <br />
* modify the config file /vzconf/<veid>.conf <br />
* rename the virt conf file <br />
* restart the new virt.<br />
<br />
that ought to do it..<br />
<br />
<br />
== Re-enable a customer account in the management system ==<br />
<br />
For example to re-instate col03438, In mysql:<br />
<br />
<pre><br />
show tables ;<br />
select * from customers where cid='col03438' ;<br />
update customers set status=1 where cid='col03438' ;<br />
select * from systems where cid='col03438' ;<br />
update systems set stop_date=NULL where cid='col03438' ;<br />
update systems set cancel_date=NULL where cid='col03438' ;<br />
update systems set status=1 where cid='col03438' ;<br />
select * from subs where cid='col03438' ;<br />
update subs set stop_date=NULL where cid='col03438' ;<br />
update subs set status=1 where cid='col03438' <br />
<br />
</pre><br />
<br />
= Ganeti VPS =<br />
<br />
This is how to create a VPS on the Lamphost ganeti cluster.<br />
I have created scripts for creating a FreeBSD VPS and one for<br />
creating a Linux VPS.<br />
<br />
== Ganeti Insall a VPS ==<br />
<br />
To create a linux VPS use<br />
<pre><br />
/root/new <name> <disk size> <memory> <IP Addr> [<OS>]<br />
<br />
( for example /root/new vps.example.com 30 2 69.55.238.10 centos-8 )<br />
</pre><br />
<br />
To create a FreeBSD VPS, we have to do a full install. This will create a <br />
new VPS.<br />
<br />
<pre><br />
/root/newfreebsd12 <name> <disk size> <memory> <IP Addr> [<OS>]<br />
</pre><br />
<br />
You will then have to do the install via VNC.<br />
<br />
You need to customize the install by adding a "jci" account and a customer<br />
account (col0XXXX) as well. <br />
<br />
<br> <br />
You will need to install sudo.<br />
<pre> <br />
pkg install sudo # install sudo package<br />
visudo # to add wheel account to sudoers<br />
vi /etc/group # add jci and user acct to wheel<br />
</pre><br />
<br />
<br><br />
You will also have to set up a console in /etc/ttys.<br />
<pre><br />
vi /etc/ttys<br />
</pre><br />
<br />
== Ganeti Re-Insall a VPS ==<br />
<br />
Re-installing the OS will destroy all data on the VPS.<br />
Login to the Ganeti master node (currently gn1).<br />
<br />
To reinstall the OS on a VPS the VPS must be shutdown.<br />
<pre><br />
gnt-instance shutdown <VPSname><br />
</pre><br />
<br />
Do the reinstall.<br />
<pre><br />
gnt-instance reinstall <VPSname><br />
</pre><br />
<br />
Restart the VPS.<br />
<pre><br />
gnt-instance startup <VPSname><br />
</pre><br />
<br />
Now you need to do the customization for the VPS. This is<br />
basically the same as the Dedicated server customization.<br />
<br />
= FreeBSD 10.1 VPS (jail5 and jail6) =<br />
<br />
This is information on how to manage the FreeBSD 10.1 VPS servers on Jail5.<br />
<br />
<br />
== Making new customer VPS ==<br />
<pre><br />
virtmake IP[,IP] <CID> hostname ipfw# email {FSP|FB|FS|FM|FP} [size in GB]<br />
</pre><br />
<br />
Note: <br />
For now, CID must be unique in the last 2 digits of the CID.<br />
<br />
Must do a "vm start <CID>" after creating a VPS.<br />
<br />
== Starting VPS ==<br />
To start all vms on the system<br />
<pre><br />
vm start<br />
</pre><br />
<br />
To start a single VPS <br />
<pre><br />
vm start <CID><br />
</pre><br />
<br />
== List running VPS ==<br />
<pre><br />
vm status<br />
</pre><br />
<br />
== Attach to a user VPS console ==<br />
<pre><br />
vm attach <CID><br />
</pre><br />
<br />
<br />
To detach from a console, type tilde control-d (~ ctl-d).<br />
<br />
== Stop a VPS ==<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
== Restart a VPS ==<br />
<pre><br />
vm restart <CID><br />
</pre><br />
<br />
<br />
== Modify a VPS ==<br />
<br />
To modify a VPS config, stop the VPS<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
Mount the VPS data file<br />
<pre><br />
vm mount <CID><br />
<br />
cd /data/vm/private/<CID>/mount<br />
</pre><br />
<br />
Modify the user files necessary.<br />
<br />
Restart the VPS.<br />
<br />
<pre><br />
vm umount <CID><br />
<br />
vm start <CID><br />
</pre><br />
<br />
== Notes on a new FBSD 10.x VPS (jail6) ==<br />
<br />
<pre><br />
#! /bin/sh<br />
#<br />
# new <colo#> <disk><br />
<br />
<br />
memory=1024<br />
<br />
x="$1"<br />
<br />
disk="$2"<br />
if [ "y" == "y$disk" ] ; then<br />
disk=60<br />
fi<br />
<br />
disk2=`expr $disk - 4`<br />
<br />
<br />
vm image provision 04dcdf96-d843-11e6-b131-001e67fcc64f col0$x<br />
# vm configure col0$x<br />
sed -e "s/memory=256M/memory=${memory}M/" /vms/col01114/col01114.conf >/vms/col0$x/col0$x.conf<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
# vm start col0$x<br />
<br />
<br />
exit<br />
<br />
vm console col0$x<br />
<br />
#<br />
# to increase the customer disk<br />
#<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
<br />
## on the customer VPS<br />
<br />
gpart recover vtbd0<br />
gpart show vtbd0<br />
swapoff /dev/vtbd0p3<br />
gpart delete -i 3 vtbd0<br />
gpart show vtbd0<br />
sysctl kern.geom.debugflags=16<br />
gpart resize -i 2 -a 4k -s $disk2\G vtbd0<br />
<br />
growfs /dev/vtbd0p2<br />
<br />
gpart show vtbd0<br />
gpart add -t freebsd-swap -a 4k vtbd0<br />
gpart show vtbd0<br />
swapon /dev/vtbd0p3<br />
<br />
df -h<br />
reboot<br />
<br />
vi /etc/rc.conf<br />
hostname<br />
IP Addr<br />
passwd root<br />
passwd user<br />
clear history<br />
cp /dev/null /var/log/utx.log<br />
cp /dev/null ~root/.history<br />
history -c<br />
<br />
</pre><br />
<br />
= FreeBSD Jails =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Making new customer jail 4.x ==<br />
<br />
1. run [[#js|js]] to figure out which partition, IP to put it on, find an unused vn, and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vnX hostname shorthost quad/safe# ipfw# email [gb disk, default 4]</tt><br />
<br />
jail14# /tmp/jailmake 69.55.226.152 /mnt/data2/69.55.226.152-col00182 vn23 negev.cerastes.org negev 4 182 cerastes@cerastes.org<br />
<br />
<br />
== Making new customer jail 6.x ==<br />
<br />
1. run [[#js|js]] to figure out which gvinum vols are free (and which mnt (disk) each gvinum goes with/to), IP to put it on and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vN[,vN] hostname shorthost quad/safe# ipfw# email</tt><br />
<br />
jail19# jailmake 69.55.236.17 /mnt/data1/69.55.236.17-col01574 v10 uncle-leo.commadev.com uncle-leo 1 1574 lee@commadev.com,lee@gmail.com<br />
<br />
<br />
== Changing an ip for freebsd VPS ==<br />
<br />
*stopjail (hostname)<br />
*on systems using vnfile or mdfile: rename vnfile/mdfile, using new ip<br />
*rename dir using new ip<br />
*Edit quad (make sure to update all lines: directory, vn/mdfile)<br />
*buildsafe (not necessary on systems having only quad1)<br />
*if you're adding an ip not already on the host machine: <tt>ipadd x.x.x.x</tt><br />
*startjail (hostname)<br />
if backups: <tt>mvbackups</tt> ...<br />
*Edit dir/ip in mgmt<br />
*if any rules, update any firewall rules to use new ip (careful not to make sure there arent multiple rules per ipfw#- search by colo#)<br />
*(if customer asks and has domains) update their domains on ns1c<br />
*optional: update ipfw counters on the host server<br />
<br />
<br />
== Rename a gconcat vol ==<br />
<br />
*stop jail, unmount everything<br />
*gconcat stop vol<br />
*gconcat clear /dev/gvinum/vN<br />
*gconcat clear /dev/gvinum/vN+1<br />
*gconcat label -v newname /dev/gvinum/vN /dev/gvinum/vN+1<br />
*bsdlabel /dev/concat/newname<br />
(make sure a partition is not fstype unused, if so change to: 4.2BSD)<br />
<br />
<br />
== Remaking a VPS (on same jail) ==<br />
<br />
# take him down (stopjail)<br />
# make note of host, vn/gvin/md, ip<br />
# remove from quad/safe<br />
# move vnfile (rename)<br />
# remove dir (jailmake will make a new one)<br />
# remove ipfw counts<br />
# jailmake<br />
# update backup script<br />
# update db with new dir/ip (use the “move” link, mark old as stopped today)<br />
# update firewall if changed ip<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
<br />
== Adding disk to a 4.x jail ==<br />
<br />
NOTE: if you unmount before vnconfig, re-vnconfig then unmount then vnunconfig<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them.<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to dump to a different partition and create the new vn on the current partition. If no space to make make dump, then move old vn’s to backup or worse, dump to backup.<br />
<br />
<br />
2. <tt>g <customerID></tt> to get the info<br />
<br />
<br />
3. make vnfile, newfs and mount. USE CAUTION not to overwrite existing vn. <br><br />
Ex: <tt>vnconfig –T -S 9g -s labels -c /dev/vn22 /mnt/data2/65.214.160.117-col00496<br><br />
disklabel -r -w vn30 auto<br><br />
newfs /dev/vn30c<br><br />
mkdir /mnt/data2/65.214.160.117-col00496-DIR<br><br />
mount /dev/vn30c /mnt/data2/69.55.237.26-col00241-DIR<br><br />
cd /mnt/data2/65.214.160.117-col00496-DIR</tt><br />
<br />
<br />
4. <tt>jailkill <hostname></tt><br />
<br />
<br />
5. <tt>dump -0a -f - /dev/vn1 | restore -r -f -</tt> <br />
<br />
<br />
6. <tt>rm restoresymtable</tt><br />
<br />
<br />
7. unmount and unvnconfig old system:<br><br />
<tt>umount /dev/vnNNc <br><br />
vnconfig –u /dev/vnNN</tt><br />
<br />
<br />
8. move vnfile. Ex: <tt>mv /mnt/data1/69.55.237.26-col00241 /mnt/data1/old-col00241-vnfile-20110312-noarchive</tt><br />
<br />
<br />
9. remove old dir: <tt>rmdir /mnt/data1/69.55.237.26-col00241-DIR</tt><br />
<br />
<br />
10. edit quad/safe to point to new location, run <tt>[[#buildsafe|buildsafe]]</tt><br />
<br />
<br />
11. start jail: <tt>[[#startjail|startjail]] <hostname></tt><br />
<br />
<br />
12. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
13. mv backups if necessary<br />
<br />
WARNING: if you restore to unmounted vn then you’re actually restoring to /mnt/data1<br />
you can tar and do above to recover<br />
<br />
NOTE: you can’t move a customer to a system that has a base sys that is diff from the one they came from (BSD 4.8 to 4.5 = won’t work)<br />
<br />
NOTE: we don’t mount procfs's to anyone’s system by default.<br />
<br />
NOTE: mount_nulls: <br />
also goofy mount_nulls seen in df -k on jail2<br><br />
also needs to be done in safe and quad<br><br />
if he wants another, use safe/quad as example and add it to file. also exec command at prompt to add mount (no reboot, remount or rejail necessary)<br />
<br />
== Moving customer to another jail machine ==<br />
<br />
Systems should only be moved between similarly-versioned jails. If you cannot get a match, you should go to freebsd.org and see what the differences were between the version you're moving from and the new version. Many of the updates will be to drivers and kernels - these don't affect the customer. Perhaps they updated traceroute or a library...you should copy the new/changed files into the VPS/jail on the new host. An alternate method for moving to a different version would be to buildworld to bring their full distribution up to date. Optional steps for this kind of upgrade are included below.<br />
<br />
1. <tt>g <customerID></tt><br />
<br />
<br />
2. <tt>[[#jailkill|jailkill]] <hostname></tt><br />
<br />
<br />
3. create new device on target system<br><br />
4.x:<br><br />
<pre>vnconfig –T –S 4g –s labels –c /dev/vn1 /mnt/data1/69.55.22x.x-col00XXX<br />
disklabel –r –w vn1 auto<br />
newfs /dev/vn1c</pre><br />
<br />
6.x:<br><br />
<pre>bsdlabel -r -w /dev/gvinum/v1<br />
newfs /dev/gvinum/v1a<br />
or <br />
gconcat label v1-v3 /dev/gvinum/v1 /dev/gvinum/v2 /dev/gvinum/v3<br />
bsdlabel -r -w /dev/concat/v1-v3<br />
newfs /dev/concat/v1-v3a</pre><br />
<br />
7.x+:<br><br />
Run jailmakeempty then skip step 4 & 7 below<br />
<br />
<br />
4. make and mount the dir on new system. <br><br />
Ex: <pre>mkdir /mnt/data2/69.55.230.3-col00123-DIR<br />
chmod 755 /mnt/data2/69.55.230.3-col00123-DIR<br />
mount [device] /mnt/data2/69.55.230.3-col00123-DIR</pre><br />
<br />
<br />
5. [[#stopjail|stopjail]] <hostname> 1<br />
<br />
<br />
6. dump fs to new system. ex: <tt>[[#dumpremoterestore|dumpremoterestore]] /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR</tt> (make sure you can ssh as root on remote machine) <br />
<br />
<br />
6a. OPTIONAL BUILDWORLD:<br><br />
cd /usr/src<br />
make world DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
cd etc<br />
make distribution DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
(you may have to rm an openssh file and re-make dist)<br />
<br />
rm -rf /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/periodic/daily/400.status-disks<br />
vi /etc/periodic/security/100.chksetuid<br />
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt><br><br />
with: <tt>MP='/' (use single quotes)</tt><br />
<br />
Ask user if they want ports overwritten with current, if yes:<br />
cp -r /usr/ports /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/usr<br />
<br />
on source: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d .<br />
(copy in all info)<br />
<br />
on target: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d . <br />
(paste all info)<br />
<br />
on source: <br />
cat /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group<br />
(copy in all info)<br />
<br />
on target:<br />
cat > /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group <br />
(paste all info)<br />
<br />
<br />
7. edit quad on source system copy over entries for the jail to the target system <br />
take care that the vn/gvinum/gconcat/md devices on the target system are’t in use and that the /mnt/dataN path doesn’t need changing<br />
<br />
<br />
8. run [[#buildsafe|buildsafe]] (if this is <=6.x) on target system (copies changes made to quad into safe file)<br />
<br />
<br />
9. remove ip from source system. Ex: <tt>ipdel 69.55.230.3</tt><br />
<br />
<br />
10. add ip to target system. Ex: <tt>ipadd 69.55.230.3</tt><br />
<br />
<br />
11. start new system, with [[#startjail|startjail]] or manually, pasting entries found from running <tt>g <customerID></tt> on the new system<br />
<br />
<br />
12. run <tt>[[#canceljail|canceljail]] col0xxxx</tt> on source system. This should prompt you to remove backups if any existed. Do not let it do this, rather:<br />
<br />
<br />
13. if backups existed move them to the new host via the <tt>[[#mvbackups_.28freebsd.29|mvbackups]] script</tt><br />
<br />
<br />
14. edit quad on source system, edit comment to reflect a move rather than a cancel, ex: <tt># moved to jail2 col00241</tt><br />
<br />
<br />
15. edit mgmt to reflect new host and dir for new system<br />
<br />
<br />
16. optional add ipfw rules on new system. Ex:<br />
<tt>ipfw add 01231 count ip from 69.55.230.2 to any<br />
ipfw add 01232 count ip from any to 69.55.230.2</tt><br />
<br />
== Increasing inodes for a VPS ==<br />
<br />
when doing the newfs: <br />
newfs -i 4096 ...<br />
<br />
<br />
== Jail services unable to contact services within same jail ==<br />
<br />
This is due to a messed up routing table (only seen it on jail2 before). <br />
<br />
route delete 69.55.228.65/32 route add -net 69.55.228.65/32 -iface fxp0 -nostatic -cloning route add 69.55.228.65 -iface lo0<br />
<br />
<br />
== /dev/null permission resets == <br />
<br />
Applies to 4.x jails only<br />
<br />
For some reason that I do not understand at all, the /dev/null node in customer jails, after the system is restarted, often reverts permissions to 0600 ... which is bad, because non-root processes cannot redirect to /dev/null anymore. In fact, a fair number of server daemons will complain and/or fail if this is the case.<br />
<br />
So, after a restart, after all systems have restarted, run postboot, which does something like:<br />
<br />
for f in `df -k | grep /dev/vn | awk '{print $6}'` ; do chmod 0666 $f/dev/null ; done<br />
<br />
Later, after the system is up, if anyone ever complains about their /dev/null permissions for any reason ... really if anyone mentions<br />
/dev/null in any way, tell them to run:<br />
<br />
chmod 0666 /dev/null<br />
<br />
We have ultimately fixed this by adding chmods to the quad/safe's<br />
<br />
<br />
== mknod in a jail == <br />
<br />
For a customer who wants to run a chroot name server (named) in a jail.<br />
<br />
<pre><br />
cd /mnt/data1/<user directory>/var/named<br />
mknod -c 1 1 null<br />
mknod -c 2 3 random<br />
</pre><br />
<br />
== Postfix Problems ==<br />
<br />
This hasn't happened in a long time, probably no longer valid with modern OS's<br />
<br />
Postfix is an alternate MTA - people replace sendmail with it in much the same way that many people also replace sendmail with qmail. The problem is that if you install postfix inside of a jail, by default it will not work properly - the user will see error messages like this:<br />
<br />
incoming mail generates errors like this:<br />
<br />
mail_queue_enter: create file incoming/298266.55902: File too large<br />
<br />
and further, sending mail generates errors like this:<br />
<br />
postdrop: warning: mail_queue_enter: create file maildrop/627930.56676: File too large<br />
<br />
This is very easy to solve. In fact, the FreeBSD welcome email that is sent out by `jailmake` now contains this block of text:<br />
<br />
- if you plan on installing postfix, email us first and ask for a necessary patch.<br />
<br />
So, anytime you see a support email that says anything about postfix not working, paste that line from the welcome email into your response to them, and right away, forward the postfix patch to them in an email.<br />
<br />
Here are the instructions:<br />
<br />
Simply copy the patch into /usr/ports/mail/postfix/files, rename the file to ‘patch-file_limit.c’ and then cd to /usr/ports/mail/postfix and run `make install`.<br />
<br />
The postfix patch is the very first email in the support email box, and we never delete or save it because we always want it there to forward to people. The instructions in the email are very clear, and it solves their problem 100% of the time.<br />
<br />
<br />
== Problems with `find` ==<br />
<br />
There was a bug in the FreeBSD vn-filesystem code. The bug causes the system to crash or hang when someone runs the `find` command inside their vn-backed filesystem (and even though the bug may have been fixed, due to the i/o overhead, we still don't like find).<br />
<br />
Now, this does not always happen - however, any jail machine with more than 20 systems on it will crash every night if every system on it runs the daily periodic script out of their crontab, and updates their locate database with the find command.<br />
<br />
(the locate database is a small, FreeBSD specific database that is populated nightly from a cron job, and is then used to provide fast answers to the `locate` command)<br />
<br />
In normal systems (32-40 jails on them), all that has to be done is make sure nobody runs the daily periodic. This means that the file /etc/crontab, instead of looking like this:<br />
<br />
<pre># do daily/weekly/monthly maintenance<br />
1 3 * * * root periodic daily<br />
15 4 * * 6 root periodic weekly<br />
30 5 1 * * root periodic monthly</pre><br />
<br />
Needs to look like this:<br />
<br />
<pre># DO NOT UNCOMMENT THESE - contact support@johncompanies.com for details<br />
#1 3 * * * root periodic daily<br />
#15 4 * * 6 root periodic weekly<br />
#30 5 1 * * root periodic monthly<br />
## DO NOT UNCOMMENT THESE ^^^^^^^^</pre><br />
<br />
and the problem will generally not occur at all. New filesystem images are always altered in this way, so all systems are set like this.<br />
<br />
However, if you have enough systems running, even casual uses of find, outside of the periodic scripts, can cause the system to crash. We are seeing this on jail13, which crashes about once per week.<br />
<br />
There is not yet any resolution to this problem.<br />
<br />
So, if you have a freebsd system crash, see if anyone has uncommented their daily periodic line - you can see them for everyone by running:<br />
<br />
/bin/sh<br />
<br />
and then running:<br />
<br />
<pre>for f in `df -k | grep vn | awk '{print $6}'` ; do echo `cat $f/etc/crontab | grep "periodic daily"` $f ; done | more</pre><br />
<br />
(all one line, one command)<br />
<br />
And then edit those /etc/crontabs and comment them out again.<br />
<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
* if you are getting disk full messages for a BSD customer, it's fairly safe to clear out their /usr/ports/distfiles dir<br />
* 4.x: ps and top an only be run by root in these jails. Done on purpose:<br />
As for `ps` and `top` - non-root users can run them, just not successfully because we have locked the permissions on /dev/mem and /dev/kmem to be root-readable only. That is why non-root users cannot successfully run `ps` and `top`.<br />
* user quotas do not work on freebsd jails - you cannot set up quotas at all, and that's that.<br />
* You cannot inject a process into a 4.x jail, only HUP running processes. 6.x and onward you can with jexec<br />
* jails see base machine’s uptime/load when running top/w<br />
* if someone is unable to get in- cant ping, etc- see if they were blocked by castle (DoS), see if their ip is on the system (post reboot it was lost cause wasn’t in the rc.conf). preboot should catch that.<br />
* in FreeBSD you cant su to root unless you belong to wheel group – so if you remove your acct and setup a new one, we have to add it to wheel group (add to /etc/group)<br />
* Dmesg from underlying sys is seen in customers dmesg on jail<br />
* Popper process going crazy @ 40% for 10min = someone who leaves mail on server<br />
* Don’t force umounts on 4.x jails – it crashes the machine, generally ok on newer machines (running md)<br />
* Good book for admin http://search.barnesandnoble.com/booksearch/isbnInquiry.asp?userid=t824VyRAYz&isbn=0596005164&itm=2<br />
* Self-sign ssl cert http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#selfcert<br />
* conversation with Glenn about semaphores and pgsql<br />
<pre>SDBoody: hey, these are valid amounts/figures/increments right:<br />
kern.ipc.semmni=1280<br />
kern.ipc.semmns=1280<br />
<br />
gr8feen: probably... I always forget exactly what those are, so I usually have to look them up<br />
SDBoody: semaphores<br />
SDBoody: i took the current 1024 and added 256 to them<br />
SDBoody: need more for pgsql<br />
gr8feen: I meant the mni and mns parts...<br />
gr8feen: some of those are not ones you just want to add to... hang on a sec and I'll look them up..<br />
gr8feen: what's semmsl set to?<br />
SDBoody: kern.ipc.semmsl: 1024<br />
<br />
SDBoody: kern.ipc.msgseg: 2048 <br />
kern.ipc.msgssz: 8 <br />
kern.ipc.msgtql: 40 <br />
kern.ipc.msgmnb: 2048 <br />
kern.ipc.msgmni: 40 <br />
kern.ipc.msgmax: 16384 <br />
kern.ipc.semaem: 16384 <br />
kern.ipc.semvmx: 65534 <br />
kern.ipc.semusz: 152 <br />
kern.ipc.semume: 10 <br />
kern.ipc.semopm: 100 <br />
kern.ipc.semmsl: 1024 <br />
kern.ipc.semmnu: 512 <br />
kern.ipc.semmns: 1024 <br />
kern.ipc.semmni: 1024 <br />
kern.ipc.semmap: 768 <br />
kern.ipc.shm_allow_removed: 0 <br />
kern.ipc.shm_use_phys: 1 <br />
kern.ipc.shmall: 262144 <br />
kern.ipc.shmseg: 256 <br />
kern.ipc.shmmni: 784 <br />
kern.ipc.shmmin: 1 <br />
kern.ipc.shmmax: 536870912 <br />
kern.ipc.maxsockets: 25600 <br />
<br />
gr8feen: ok...msl is max per id, mni is max ids, mns is max number of semaphores... so you probably want something like mns = mni * msl<br />
gr8feen: which one did you run out of?<br />
SDBoody: not sure how to tell- ipcs shows the sems in use add up to 1024<br />
SDBoody: there are 59 entries<br />
gr8feen: I'm assuming you tried to start postgres and it failed?<br />
SDBoody: yes<br />
gr8feen: it should have logged why, somewhere..<br />
gr8feen: if I recall, it'll tell you which one it ran out of<br />
SDBoody: > DETAIL: Failed system call was semget(1, 17, 03600).<br />
<br />
gr8feen: so it wanted an id with 17 semaphores...I'd start by making mns = 17*mni and leave mni and mnl set to what they are now and see what it does<br />
SDBoody: i think mni is plenty high<br />
SDBoody: ok, more reasonable 17408<br />
gr8feen: yeah...just change that one and see how it goes..<br />
SDBoody: and leave mni alone at 1024?<br />
gr8feen: yeah...mni id the max number of ids...but if it's trying to get something like 17 per id, your going to hit mns before you hit anything else<br />
SDBoody: right, but doesn't hurt to have it that high (assuming)<br />
<br />
gr8feen: not really... I think those get allocated out of ram that you cant page out, but it's still such a small amount that it really doesn't matter<br />
SDBoody: looks like that worked, thx!<br />
gr8feen: cool<br />
<br />
see semaphores: ipcs -a -s</pre><br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
== Making new customer VE (vm) ==<br />
<br />
This applies only to new virts >= 4.x<br />
<br />
grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.<br />
<br />
1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
also, ping the IP address.<br />
<br />
2. put CT on whichever partition has more space<br />
df -h<br />
<br />
3. vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> ) <br />
vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@johncompanies.com centos-6-x86_64 LM<br />
<br />
4. copy veid, dir, ip and password to pending customer screen. activate customer<br />
<br />
== Making new customer VE (vemakexxx) ==<br />
<br />
This applies to older virts with old templates. This should probably not be used at all anymore.<br />
<br />
<br />
1. look thru hist for ip<br />
<br />
2. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
<br />
3. put ve on whichever partition has more space<br />
vemakerh9 <veid> <ip> <hostname> <mount> <email> [gb disk]; <256|384|512> <veid><br />
vemakerh9 866 69.55.226.109 ngentu.com /vz1 ayo@ngantu.com,asd@asd.com 5; 256 866<br />
<br />
4. copy (veid), dir, and ip to pending customer screen (pass set to p455agfa)<br />
<br />
<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrateonline/migrate failures (migrate manually) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
You may also see migration failing due to quota issues.<br />
<br />
You can try to resolve by copying any quota file into the file you need:<br />
<br />
cp /var/vzquota/quota.1 /var/vzquota/quota.xxx<br />
<br />
If it complains about quota running you should then be able to stop it<br />
<br />
vzquota off xxxx<br />
<br />
If all else fails, migrate to a new VEID<br />
i.e. 1234 becomes 12341<br />
<br />
<br />
If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
'''Note:''' don't forget to fix /usr/local/sbin/backup.config for new disk location<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
=== Cant change file /etc/sysconfig/network ===<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
=== What template is user using ===<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
=== Edits needed to run java ===<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
=== Set time zone ===<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
=== Failing shm_open calls ===<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
=== Debian can't get on the network ===<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
=== CentOS can't get on the network ===<br />
<br />
When CentOS system was running, but can't get on the network, they probably updated iproute via yum.<br />
<br />
Copy an older version of the iproute rpm to their VPS.<br />
<br />
<pre>cp -p /root/iproute/iproute-2.6.32-23.el6.<i686>.rpm /vz2/private/<1527>/fs/root</pre><br />
<br />
Enter their ve, install rpm, restart network, and prevent yum from updating iproute.<br />
<br />
<pre><br />
ve <1527><br />
rpm -i --force iproute-2.6.32-23.el6.<i686>.rpm<br />
rpm -q iproute<br />
rpm -e <new iproute rpm><br />
service network restart<br />
echo "exclude=iproute" >>/etc/yum/ip-exclude<br />
echo "include=/etc/yum/ip-exclude" >>/etc/yum.conf<br />
</pre><br />
<br />
<br />
<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
----<br />
<br />
* uptime/load that vz users see isn’t what we see- it's their own load<br />
* Vz overwrites resolve.conf on every boot to what’s in ve config file. Also overwrites hostname<br />
* Quotas in vz: set Gid_limit in vz config file and restart ve. Errors leading up to this will be out of files but no inode or space issues. set to 2000 in conf, and restart vz<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
To create a new Linux VPS<br />
<br />
* get an IP address from mgmt screen<br />
* run df -h to figure out which disk has enough space<br />
* run vm with no arguments to get syntax and exact template name<br />
* use col0xxxx for hostname if they don’t give you a hostname<br />
* copy over dir, veid, ip and password to pending customer screen<br />
<br />
vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> )<br />
<br />
== disable ==<br />
To disable a customers Virtuozzo VPS server.<br />
<br />
* stop the server<br />
vzctl stop <veid><br />
* prevent restart on server reboot and user restart via Control Panel<br />
vzctl set <veid> --disabled=yes --offline_management=no --save<br />
<br />
== cancelve ==<br />
When a customer cancels all service.<br />
<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
<br />
display the contents of the vz config file for the <veid><br />
<br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
<br />
List all virtual environments running on this machine.<br />
<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
<br />
show processes running under this <veid>.<br />
<br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
<br />
Restart a <veid>.<br />
<br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
<br />
enter a <veid> as root.<br />
<br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
<br />
execute a single command in the <veid> environment.<br />
<br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem VEID <256|512|768|1024|1280|1536|2048> [burst GB]<br />
adjusts the memory resources for the VE. If this is on VZ ver >4, you will also be prompted to provide a burst ram amount. Sample usage:<br />
<br />
<pre># setmem 21671 1024 4<br />
/usr/sbin/vzctl set 21671 --ram 1024m --swap 3072m --save<br />
</pre><br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>75.80.128.147https://wiki.jcihosting.com/index.php?title=New_Signups&diff=2221New Signups2020-04-02T22:20:45Z<p>75.80.128.147: /* Dedicated Server Install Checklist */</p>
<hr />
<div>__TOC__<br />
TODO- review<br />
<br />
= Open source discount signups =<br />
<br />
URLS to give to customers:<br />
https://secure.johncompanies.com/signup/step1.html?svc=bsd-oss<br />
<br />
https://secure.johncompanies.com/signup/step1.html?svc=linux-oss<br />
<br />
= New Signups = <br />
New customers sign up for service on our web based forms at<br />
www.johncompanies.com <br />
<br />
When a signup occurs an email is sent to support@johncompanies.com <br />
and signups get written to two files:<br />
<br />
/usr/local/www/jc_pub/data/pending<br />
<br />
and<br />
<br />
/usr/local/www/jc_pub/data/log/log.pending<br />
<br />
The second file is simply a running log of signups, and should not be used for anything. The only time it is handy is if you are editing the pending file, then save it, and are told that the file has changed since you started editing it. This means that someone signed up while you were editing the pending file. What you should do is force the save (thus losing their signup) and then cut and paste the lost signup from /htdocs/colocation/data/log/log.pending.<br />
<br />
Lines in the pending file look like this:<br />
<br />
<pre>2005-11-08;newcastle;Frederick Wilson;;1405 South Adams;Fort Worth;Texas;76104;US;referred by nmrc.org;LM-1;2;Limit 10;1;207.13.31.48;106<br />
vaifan@airmail.net;817-798-8637;on;on;<br />
new;add2;;;replace;replace;preserve;elaine.commadev.com</pre><br />
<br />
The fields are defined as follows:<br />
<br />
<pre>signup date;hostname;last, first;company; address;city;state;zip;country;referred by;package id;bandwidth overage option;if gobut is selected, $ amount to limit is listed here;payment method;source ip;template id;<br />
email;phone #;admin;billing;alt<br />
new/replace system;create new account/add to existing;hostname of server being replaced;new IP pref;update/merge contacts;update/merge owner info;update/preserve traffic overage;</pre><br />
<br />
When a customer signs up an email will be sent to support@johncompanies.com containing the log entry in the format above<br />
<br />
The process for setting up a new customer is as follows:<br />
<br />
1. Go to the pending customers screen (choose Mgmt site -> New Signups). New customers will be at the bottom of the "VPS Signups" and "Colo Signups" sections. If they’ve paid automatically (via PayPal) the word “PAID” will appear in the “pmt status” column. The other thing you’ll see in this column is “paid- not cleared”- this means they paid with eCheck (via PayPal) and the check has not cleared. This normally takes 3 biz. days and we wait till it clears and shows “PAID” before setting up the server. Customers who pay with a credit card will not appear as "PAID" here. In that case, an email will be sent to support@johncompanies.com indicating a payment has been made (or failed). Once the payment has been paid/cleared, click “process….” <br />
<br />
Before setting someone up, you need to decide whether the signup is fraudulent. Examples of fraud can be found in /usr/local/www/jc_pub/data/fraud-examples <br />
Generally, <br />
* NOT FRAUD = non-anonymous email accounts that match who they are (dan@brockman.com, where the person is Dan Brockman)<br />
* NOT FRAUD = referred by a customer that actually exists or by kuro5hin<br />
* MAYBE FRAUD = referred by ‘web search’ or ‘google’<br />
* MAYBE FRAUD = generic sounding address<br />
* MAYBE FRAUD = IP address is out of country and address is in US (use dnsstuff link on main signups screen to lookup where IP is)<br />
* FRAUD = they don’t type in: referred by and/or hostname and/or give the default answer for bandwidth is stop<br />
* FRAUD = multiple orders each from the same IP and/or using same email address<br />
* FRAUD = no hostname or hacker-sounding hostname: 3v1l0n3 (evilone)<br />
<br />
If you’re ever in doubt, call the card owner using the phone number given at signup or call the bank number provided for the card. Never use the email as it’s likely the thief’s and not the cardholder’s.<br />
<br />
Almost all the info needed is pre-entered into this screen, but some fields will need attention:<br />
<br />
* System: The system is already selected based on the template/OS the customer selected. <br />
* Directory: information provided for you to enter once the system is created. (leave bank for managedcolo)<br />
* Disk: indicates how much disk space the server should have. (leave bank for managedcolo)<br />
* Hostname: self-explanatory. <br />
* veid: (linux systems only) should be filled in with the significant digits of the customer ID, ex: col01340 = veid 1340. <br />
* os: indicates which OS the customer wants (feed to linux vm script) <br />
* ip(s): for linux customers, you should click “ipmap” and scroll down to the first available (green) ip amongst others on the same system. Click on the ip to copy it back to the “ip(s)” field on the form. For FreeBSD customers, the ips available for use are already assigned to the system and should be copied back to this screen once the jail is made (see below). For colo customers, choose an IP appropriate to the data center. i.e. for i2b pick an IP from the 229 block.<br />
* start date: should reflect the day the system was created. <br />
* asset tag: colo only<br />
* password: VPS only<br />
* cabinet: colo only<br />
* ats port: colo @ i2b only<br />
* monitored: should only be checked if the system’s ip/services were supplied to castle to place on the monitor (probes) list. <br />
<br />
NOTE: you must use a JavaScript enabled browser to enter new customers otherwise you won’t be given correct options for os templates.<br />
<br />
3. create the new system using either [[VPS_Management#jailmake|jailmake]] or [[VPS_Management#vm|vm]] scripts. For posterity, on our older linux systems and older OS's (pre virt17) we used to use a custom script for each OS version: vemakecentos3 vemakedebian40 vemakedebian30 vemakedebian31 vemakefedora2 vemakefedora6 vemakefedora7 vemakesuse100 vemakesuse93 vemakecentos4 vemakedebian31 vemakefedora4 vemakerh9 vemakeubuntu5 vemakefedora vemakefedora5 vemakesuse vemakeubuntu606 vemakeubuntu610 vemakeubuntu704<br />
<br />
[[VPS_Management#jailmake|jailmake]] and [[VPS_Management#vm|vm]] both email the new customer their welcome email.<br />
<br />
When you are done adding a customer, for both systems: copy back the dir and password (supplied by the VPS make script) into the form. For FreeBSD, copy back the IP which the make script will give back to you.<br />
<br />
When you are done filling out all the fields in the pending customer form, click “Activate”. This will create the customer in our database and remove them from the pending list. Nothing is emailed to the customer as a result of this action. If the customer paid via credit card, their info needs to be added manually.<br />
<br />
Discussion about choosing IPs ...<br />
<br />
When you make a new system, you have to choose what IP to give it. New FreeBSD servers are configured with a set amount of IPs, and as you add new systems to that freebsd server you can use one of the IPs assigned to the host, but not currently assigned to a VPS. You can use the js program to see what IPs are available for assigning to new customers. <br />
<br />
However, it is not that simple with the linux systems. The linux servers do not get the IPs of their customer systems bound to the actual machine. That is, even on a fully loaded linux system, if you run `ifconfig -a` from the base machine, you only see one IP - the main IP of that system. Further complicating matters is that linux systems can bind multiple IPs - therefore it is not possible to know that the next new linux system should just have the next IP as the last new one that was created.<br />
<br />
So, what we do is, new linux systems are simply assigned a starting IP, and no new machine (freebsd or linux) is assigned a base IP anywhere within 92-128 Ips of that IP. So you have 92-128 IPs to grow with for the linux systems that will live on that machine.<br />
<br />
So, you simply start creating systems from that starting point – and additional IPs that those systems need should also be added from that range, and even other linux systems on other virt machines should get new IPs from that block of IPs. To find the block of IPs in use, go thru bash history to find last ip used or assigned to a ve. Then, click the “ipmap” link (or pull up Mgmt. -> Reference -> IP Map). Ip’s not already assigned to a machine will show up as green. In general, we want to use ips which have never been assigned, or were relenquishsed years ago (or at least 30 days ago – IMPORTANT!!). <br />
D<br />
If a customer has selected an IP package – is paying for extra IPs now, then we’re pretty much obliged to assign all those IPs now. Otherwise, we only ever assign 1 IP.<br />
<br />
= The Welcome Emails =<br />
<br />
[[VPS_Management#jailmake|jailmake]] and [[VPS_Management#jailmake|vm]], as one of its arguments, asks for an email address. This email address is used to send a welcome email to the new user. However, the welcome email is not inside the jailmake/vemake script, and they are not on the jails themselves either. Here are all the various welcome emails we have:<br />
<br />
<pre>/usr/local/www/jc_pub/data/welcome-freebsdp (dynamic freebsd welcome email, 7.x 8.x)<br />
/usr/local/www/jc_pub/data/welcome-linux (dynamic linux welcome email)<br />
/usr/local/www/jc_pub/data/welcome-debian (dynamic debian/ubuntu welcome email)<br />
/usr/local/www/jc_pub/data/welcome-fedora (dynamic fedora/centos welcome email)<br />
/usr/local/www/jc_pub/data/welcome-freebsd (original, generic freebsd welcome email)<br />
/usr/local/www/jc_pub/data/welcome-freebsd6 (for freebsd6)<br />
/usr/local/www/jc_pub/data/welcome-freebsd7z (for freebsd7 with zfs DEPRECATED)</pre><br />
<br />
on the main johncompanies server.<br />
<br />
When the jailmake script is run, it issues the `fetch` (or wget) command to retrieve it, i.e.<br />
http://www.johncompanies.com/colocation/data/welcome-freebsdp<br />
<br />
and saves it as a temp file, mails it off.<br />
<br />
Welcome emails not noted as dynamic above, are emailed with the IP appended at the top to the email address specified. The password to these accounts was/is generic- not very good. Welcome emails noted as dynamic are parsed by the make script to include the IP and a generated, random root password. <br />
<br />
IMPORTANT NOTE: when creating an older-OS ve on a virt- this almost never happens (i.e. it’s created using the old-style vemakexxxx command), you must use the support address as the email address as the make script will be pulling down the dynamic welcome email when it expects to see the old, generic format and it won’t look good. When you receive the welcome email in support, format properly and resend to customer.<br />
<br />
This means three things:<br />
<br />
a) In order for jailmake to work, the johncompanies web server needs to be up and running, and that file needs to exist at that URL<br />
<br />
b) If you want to edit the welcome email, you only need to edit it on the web server, in one place<br />
<br />
c) If a customer for some reason does not receive the welcome email, then you need to go Mgmt -> Reference -> (file), copy and paste it into an email to them. For the old-generic emails, make sure to add this line to the very top:<br />
<br />
IP: (their IP)<br />
<br />
If the welcome email does not exist, jailmake will still work, in the sense that it will create the system, but no welcome email will be sent.<br />
<br />
= Dedicated Server Setup =<br />
<br />
Before starting the OS install you'll need to know the following pieces of info:<br />
<br />
*CID:<br />
*Customer has multiple colos: Y/N<br />
*Asset tag: JC-xxxx<br />
*rack/location: <br />
*Service/Package (and any deviations to B/W, nfs space, IPs, etc) and price:<br />
*RAM in system:<br />
*IPs included in plan:<br />
*OS (32 vs 64bit variant):<br />
*Hostname:<br />
*Disk partitions, including swap space:<br />
*Number of initial IPs to assign:<br />
*Timezone:<br />
<br />
Some to most of that info will be provided via the new signup page, assuming the customer ordered the server via our order page. The rest will comes from the sales/build department.<br />
<br />
<br />
Once the server is built, it should be installed in the rack, booted to the BIOS screen and labeled with the asset tag and the customer's CID (if available/established).<br />
<br />
If an [[IPKVM]] is not already attached, you will need to ascertain which one is available (look in ~user/kvm, usually open for editing in the p4 screen of the mailbox window).<br />
<br />
Before loading the server, there's a couple things to do in the BIOS screen:<br />
# make sure the date and time are set to UTC<br />
# make sure the server's power restore action is set to: last state (basically what we're trying to do here is if the server has power pulled, when power is restored it should turn back on- assuming it was on when power was pulled. This is what allows our ATS power cycling to work. If this is not set, when the ATS port is turned off and turned back on, the server may not turn back on.<br />
# set the boot order to disable network booting or other things that may slow down bootup. <br />
<br />
After saving and exiting the BIOS, if the server has a raid card installed, you will see the raid BIOS screen. You should enter this screen and setup a raid mirror, or whatever the customer has requested (if a special request was made sales will let you know). Usually we use a 3ware card, to enter the config screen, press ALT-3. Use spacebar to select the 2 drives, tab to 'create unit'. Use defaults. Create a raid1 mirror (unless customer asks for something else). Do not enable write cache, assuming no battery exists. F8 to save and exit.<br />
<br />
Reboot and load the OS- follow the instructions for pulling the ISO into the IPKVM and booting to it [[IPKVM|here]].<br />
<br />
The install should be for the server version of whatever OS was requested. We typically do not install anything other than an sshd (and ports (tree) for FreeBSD). We don't install a GUI environment. We do not setup auto updates. We do not encrypt home directories. <br />
<br />
When selecting and IP, take into consideration the data center: an install at i2b should use an IP from the IPs routed to i2b, and vice versa. A customer's package may come with multiple IPs, however unless they indicate they want more assigned initially (and show good reason for doing so) we only assign 1 IP. If they are to receive more IPs, you will/may need to configure those post-install (depending on the OS). Ideally all IPs they receive are on the same class C block.<br />
<br />
When picking a NIC (most of the installs we do are network versions and require the network to download OS components), it's somewhat of a crap shoot as to which NIC to choose and configure. Usually we try to begin pinging the IP we assign and watch for it to respond immediately after configuring the NIC. If it does not ping, you can:<br />
# ask the NOC to swap the network cable to the alternate port<br />
# go back and configure the alternate NIC (you may have to restart the OS install if it doesn't properly unconfigure the initial NIC, or you can't change it's config to alleviate an IP conflict)<br />
<br />
We set DNS: 69.55.229.3, 69.55.225.225 (for a server @ i2b), 69.55.225.225, 69.55.230.3 (for a server @ castle)<br />
<br />
We setup a regular user account: 'user' and we set that password to 'newrootNNNN' where NNNN is the sig digits of the customer's CID. i.e. for col01233 the password is 'newroot1233'. In FreeBSD, make sure to add 'user' to the wheel group (member of). <br />
If a root password is allowed to be set, we use the same password: 'newrootNNNN' <br />
<br />
Once the OS is installed and you've rebooted (after disconnecting the ISO from the KVM) you should confirm all is working: swap, ram, disk partitions, network is as it should be. If you were only able to assign 1 DNS server in setup, please add another to /etc/resolv.conf:<br />
<br />
@i2b: 69.55.229.3, 69.55.225.225<br />
@castle: 69.55.225.225, 69.55.230.3<br />
<br />
Assuming they have a 3ware raid card, you will need to transfer in the raid CLI tool and our handy script so the customer may check on the health of their raid array, as instructed/encouraged by the welcome email they will receive.<br />
<br />
This package is located on mail;<br />
<pre><br />
fetch http://johncompanies.com/tools/3ware/L64.tgz<br />
tar xvzf L64.tgz<br />
</pre><br />
<br />
The version you choose will of course depend on the OS/version you're installing. You should copy the tarball to the /usr/local/sbin dir (or something in their path) and untar'd. You should run the script included in the tarball called checkraid.sh. If it gives an error, this is due to the fact that on some servers the raid card is recognized as 'c1' and on others 'c0' or something different. Determine which it is for this server by running:<br />
tw_cli info<br />
<br />
which will give you output like:<br />
<pre>Ctl Model (V)Ports Drives Units NotOpt RRate VRate BBU<br />
------------------------------------------------------------------------<br />
c1 9650SE-8LPML 8 6 1 0 5 1 -<br />
</pre><br />
<br />
In this example it's 'c1' so edit checkraid.sh and change all instances of 'c0' to 'c1'<br />
<br />
If we're adding additional IPs (and were not able to do so during initial install), configure as follows:<br />
<br />
Centos/Fedora:<br />
<br />
<pre><br />
cd /etc/sysconfig/network-scripts/<br />
cp -p ifcfg-eth* ~<br />
fetch http://johncompanies.com/tools/network/centos/ifcfg-eth.tgz<br />
tar xvzf ifcfg-eth.tgz<br />
<br />
vi ifcfg-eth0:<br />
<br />
DEVICE=eth0<br />
TYPE=Ethernet<br />
ONBOOT=yes<br />
NM_CONTROLLED=yes<br />
BOOTPROTO=none<br />
IPADDR=69.55.231.XXX # change this to main IP address<br />
PREFIX=24<br />
GATEWAY=69.55.231.1 # change this to 229.1 if IP addr in other block<br />
DNS1=69.55.229.3<br />
DNS2=69.55.225.225<br />
DNS3=69.55.230.3<br />
DOMAIN="johncompanies.com"<br />
DEFROUTE=yes<br />
IPV4_FAILURE_FATAL=yes<br />
IPV6INIT=no<br />
NAME="System eth0"<br />
</pre><br />
<br />
<pre><br />
vi eth0:*<br />
<br />
DEVICE=eth0:0<br />
TYPE=Ethernet<br />
ONBOOT=yes<br />
NM_CONTROLLED=yes<br />
BOOTPROTO=none<br />
IPADDR=69.55.231.XXX # change this to IP address<br />
PREFIX=24<br />
IPV4_FAILURE_FATAL=yes<br />
IPV6INIT=no<br />
</pre><br />
<br />
Repeat above as needed for each IP address<br />
<br />
Remove the un-needed files.<br />
<pre><br />
rm ifcfg-eth0:[3-8] # change numbers to fit how many IP addresses are needed<br />
</pre><br />
<br />
Debian/Ubuntu:<br />
<br />
<pre><br />
cd /etc/network<br />
vi interfaces<br />
<br />
Old file was:<br />
# The loopback network interface<br />
auto lo<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
auto eth0<br />
iface eth0 inet static<br />
address 69.55.227.4<br />
netmask 255.255.255.0<br />
network 69.55.227.0<br />
broadcast 69.55.227.255<br />
gateway 69.55.227.1<br />
# dns-* options are implemented by the resolvconf package, if installed<br />
dns-nameservers 69.55.225.225<br />
dns-search yeppernet.com<br />
<br />
new file is:<br />
# The loopback network interface<br />
auto lo<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
auto eth0<br />
iface eth0 inet static<br />
address 69.55.227.44<br />
netmask 255.255.255.0<br />
network 69.55.227.0<br />
broadcast 69.55.227.255<br />
gateway 69.55.227.1<br />
# dns-* options are implemented by the resolvconf package, if installed<br />
dns-nameservers 69.55D.225.225<br />
dns-search yeppernet.com<br />
<br />
auto eth0:0<br />
iface eth0:0 inet static<br />
address 69.55.227.55<br />
netmask 255.255.255.0<br />
network 69.55.227.0<br />
broadcast 69.55.227.255<br />
gateway 69.55.227.1<br />
<br />
</pre><br />
/etc/init.d/networking restart<br />
(make sure you have console in case you screw that up)<br />
<br />
<pre>ifconfig<br />
eth0 Link encap:Ethernet HWaddr 00:30:48:28:9d:50<br />
inet addr:69.55.227.44 Bcast:69.55.227.255 Mask:255.255.255.0<br />
inet6 addr: fe80::230:48ff:fe28:9d50/64 Scope:Link<br />
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br />
RX packets:2850149 errors:0 dropped:0 overruns:0 frame:0<br />
TX packets:2595566 errors:0 dropped:0 overruns:0 carrier:0<br />
collisions:0 txqueuelen:100<br />
RX bytes:1193068127 (1.1 GB) TX bytes:2046984105 (1.9 GB)<br />
Base address:0x3000 Memory:fc400000-fc420000<br />
<br />
eth0:0 Link encap:Ethernet HWaddr 00:30:48:28:9d:50<br />
inet addr:69.55.227.55 Bcast:69.55.227.255 Mask:255.255.255.0<br />
inet6 addr: fe80::230:48ff:fe28:9d50/64 Scope:Link<br />
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br />
RX packets:2850149 errors:0 dropped:0 overruns:0 frame:0<br />
TX packets:2595566 errors:0 dropped:0 overruns:0 carrier:0<br />
collisions:0 txqueuelen:100<br />
RX bytes:1193068127 (1.1 GB) TX bytes:2046984105 (1.9 GB)<br />
Base address:0x3000 Memory:fc400000-fc420000<br />
</pre><br />
<br />
Ubuntu 18.04<br />
<br />
Ubuntu 18.04 uses netplan instead of /etc/network/interfaces.<br />
<br />
<pre><br />
vi /etc/neplan/50-cloud-init.yaml<br />
<br />
# This file is generated from information provided by<br />
# the datasource. Changes to it will not persist across an instance.<br />
# To disable cloud-init's network configuration capabilities, write a file<br />
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:<br />
# network: {config: disabled}<br />
network:<br />
version: 2<br />
renderer: networkd<br />
ethernets:<br />
enp0s25:<br />
dhcp4: no<br />
addresses: [69.55.229.26/24]<br />
gateway4: 69.55.229.1 <br />
nameservers:<br />
addresses: [69.55.229.3,8.8.8.8,8.8.4.4]<br />
<br />
netplan apply<br />
</pre><br />
<br />
The very last thing you should do before logging off the server is to run:<br />
history -c<br />
in every shell you were in so they don't see what we were doing :)<br />
<br />
Assuming this server is connected to an ATS, you'll want to confirm the server is connected to the port you think it is, and test the functionality- that you can power cycle the port and the server will come back up when power is restored. You should do a test power cycle (via the mgmt or AM-based ATS control) while the server is in post or BIOS. Basically, just NOT while the OS is running and disks mounted.<br />
<br />
You should confirm the switch ports to which the server is connected (maybe has a 2nd port for the RMM). While the server is rebooting during your ATS test, you can observe output on the switch console that looks like:<br />
<pre>.Mar 7 09:47:45 PST: %LINK-3-UPDOWN: Interface FastEthernet0/14, changed state to down<br />
.Mar 7 09:47:46 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/14, changed state to down<br />
.Mar 7 09:47:47 PST: %LINK-3-UPDOWN: Interface FastEthernet0/14, changed state to up<br />
.Mar 7 09:47:48 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/14, changed state to up<br />
</pre><br />
This confirms the server is on port 14.<br />
<br />
Take this info (ATS, switch ports plus the asset tag) and:<br />
# activate (or add) the server. Update the cabinet, ATS and port, asset tag. (Switch port not kept in mgmt)<br />
# update the [[cabinetmap|cabinet map]] to add the server to the correct location, along with the ATS port, and switch port(s)<br />
# login to the switch serial console and add the customer's CID (and hostname if they have mult. servers) to the port(s) description tag. Repeat for the RMM port if they have one:<br />
<pre>switch-p25#conf t<br />
Enter configuration commands, one per line. End with CNTL/Z.<br />
switch-p25(config)#int fa0/10<br />
switch-p25(config-if)#des col01233 myhostname<br />
switch-p25(config)#int fa0/11<br />
switch-p25(config-if)#des col01233 myhostname (RMM)<br />
switch-p25(config-if)#end<br />
switch-p25#<br />
.Mar 7 11:37:17 PST: %SYS-5-CONFIG_I: Configured from console by console<br />
switch-p25#wr me<br />
Building configuration...<br />
[OK]<br />
switch-p25#<br />
</pre><br />
<br />
At this point it's safe to hand the server over to the customer. Use the 'new colo welcome' paste. Edit for:<br />
# IP(s)<br />
# if it's FreeBSD use the 'root' & 'user' password line, if it's Ubuntu use/edit the 'root' password line, depending on if you set the root password or not, or what you setup for the normal user account<br />
# the correct DNS depending on which data center they're at (remove ns3c if the server is at castle)<br />
# if they have no raid card, remove that section<br />
# if they have an RMM, include the RMM paste somewhere in the email<br />
<br />
If they're already a customer, you can exclude the info about their AM login info (since they have it already). If they're not, you'll need to reset their AM pass to see it.<br />
<br />
Lastly, assuming this server was installed for a new customer and the customer was not yet activated, the CID was probably not established and the server has no CID label. You'll need to make a note to label it (and add the hostname too if they have multiple servers) next time you're at the data center, or ask the noc staff to do it for you.<br />
<br />
== Dedicated Server Install Checklist ==<br />
<br />
o Install OS<br />
<br />
o Add JCI - adduser, wheel, visudo<br />
<pre><br />
adduser -u 955 jci<br />
passwd jci<br />
''l4mph0st''<br />
adduser -u 1000 col0XXX<br />
passwd col0XXXX <br />
''col0XXXX'' (XXXX is their col0XXXX number)<br />
vi /etc/group<br />
( add "jci,user" to group wheel and/or sudo )<br />
visudo<br />
( enable group wheel to do anything with password )<br />
</pre><br />
<br />
o Install openssh-clients (if not already installed)<br />
<pre><br />
yum -y install openssh-clients<br />
</pre><br />
<br />
o Update OS<br />
<pre><br />
yum -y update (for CentOS)<br />
</pre><br />
<br />
o checkraid.sh<br />
As noted above install our RAID disk checks<br />
<br />
o config IP addresses<br />
<br />
o clear last logins<br />
<pre><br />
cp /dev/null /var/log/wtmp<br />
</pre><br />
<br />
o clear history<br />
<pre><br />
cp /dev/null ~/.bash_history<br />
history -c<br />
shutdown -h now<br />
</pre><br />
<br />
o For a suspected spammer, add rule to firewall2 to limit output for mail ports (ie.)<br />
before rule 50<br />
<pre><br />
ipfw add 37 pipe 8 ip from 69.55.231.104/30 to any dst-port 25,465,587<br />
ipfw add 37 pipe 8 ip from 69.55.231.108 to any dst-port 25,465,587<br />
</pre><br />
<br />
=== After installed in rack ===<br />
<br />
check ssh<br />
label switch port<br />
update cabinet map<br />
update mgmt cabinet number<br />
<br />
== RMM Setup ==<br />
<br />
The RMM must be configured via the server's BIOS screen:<br />
<br />
Server management > BMC LAN config:<BR><br />
down to Intel RMM3 LAN config: <BR><br />
IP (static) , mask, g/w<BR><br />
<br />
user config: <BR><br />
disable anonymous. <BR><br />
root: set pass to newrootNNNN</div>75.80.128.147https://wiki.jcihosting.com/index.php?title=New_Signups&diff=2220New Signups2020-03-29T19:01:32Z<p>75.80.128.147: /* Dedicated Server Install Checklist */</p>
<hr />
<div>__TOC__<br />
TODO- review<br />
<br />
= Open source discount signups =<br />
<br />
URLS to give to customers:<br />
https://secure.johncompanies.com/signup/step1.html?svc=bsd-oss<br />
<br />
https://secure.johncompanies.com/signup/step1.html?svc=linux-oss<br />
<br />
= New Signups = <br />
New customers sign up for service on our web based forms at<br />
www.johncompanies.com <br />
<br />
When a signup occurs an email is sent to support@johncompanies.com <br />
and signups get written to two files:<br />
<br />
/usr/local/www/jc_pub/data/pending<br />
<br />
and<br />
<br />
/usr/local/www/jc_pub/data/log/log.pending<br />
<br />
The second file is simply a running log of signups, and should not be used for anything. The only time it is handy is if you are editing the pending file, then save it, and are told that the file has changed since you started editing it. This means that someone signed up while you were editing the pending file. What you should do is force the save (thus losing their signup) and then cut and paste the lost signup from /htdocs/colocation/data/log/log.pending.<br />
<br />
Lines in the pending file look like this:<br />
<br />
<pre>2005-11-08;newcastle;Frederick Wilson;;1405 South Adams;Fort Worth;Texas;76104;US;referred by nmrc.org;LM-1;2;Limit 10;1;207.13.31.48;106<br />
vaifan@airmail.net;817-798-8637;on;on;<br />
new;add2;;;replace;replace;preserve;elaine.commadev.com</pre><br />
<br />
The fields are defined as follows:<br />
<br />
<pre>signup date;hostname;last, first;company; address;city;state;zip;country;referred by;package id;bandwidth overage option;if gobut is selected, $ amount to limit is listed here;payment method;source ip;template id;<br />
email;phone #;admin;billing;alt<br />
new/replace system;create new account/add to existing;hostname of server being replaced;new IP pref;update/merge contacts;update/merge owner info;update/preserve traffic overage;</pre><br />
<br />
When a customer signs up an email will be sent to support@johncompanies.com containing the log entry in the format above<br />
<br />
The process for setting up a new customer is as follows:<br />
<br />
1. Go to the pending customers screen (choose Mgmt site -> New Signups). New customers will be at the bottom of the "VPS Signups" and "Colo Signups" sections. If they’ve paid automatically (via PayPal) the word “PAID” will appear in the “pmt status” column. The other thing you’ll see in this column is “paid- not cleared”- this means they paid with eCheck (via PayPal) and the check has not cleared. This normally takes 3 biz. days and we wait till it clears and shows “PAID” before setting up the server. Customers who pay with a credit card will not appear as "PAID" here. In that case, an email will be sent to support@johncompanies.com indicating a payment has been made (or failed). Once the payment has been paid/cleared, click “process….” <br />
<br />
Before setting someone up, you need to decide whether the signup is fraudulent. Examples of fraud can be found in /usr/local/www/jc_pub/data/fraud-examples <br />
Generally, <br />
* NOT FRAUD = non-anonymous email accounts that match who they are (dan@brockman.com, where the person is Dan Brockman)<br />
* NOT FRAUD = referred by a customer that actually exists or by kuro5hin<br />
* MAYBE FRAUD = referred by ‘web search’ or ‘google’<br />
* MAYBE FRAUD = generic sounding address<br />
* MAYBE FRAUD = IP address is out of country and address is in US (use dnsstuff link on main signups screen to lookup where IP is)<br />
* FRAUD = they don’t type in: referred by and/or hostname and/or give the default answer for bandwidth is stop<br />
* FRAUD = multiple orders each from the same IP and/or using same email address<br />
* FRAUD = no hostname or hacker-sounding hostname: 3v1l0n3 (evilone)<br />
<br />
If you’re ever in doubt, call the card owner using the phone number given at signup or call the bank number provided for the card. Never use the email as it’s likely the thief’s and not the cardholder’s.<br />
<br />
Almost all the info needed is pre-entered into this screen, but some fields will need attention:<br />
<br />
* System: The system is already selected based on the template/OS the customer selected. <br />
* Directory: information provided for you to enter once the system is created. (leave bank for managedcolo)<br />
* Disk: indicates how much disk space the server should have. (leave bank for managedcolo)<br />
* Hostname: self-explanatory. <br />
* veid: (linux systems only) should be filled in with the significant digits of the customer ID, ex: col01340 = veid 1340. <br />
* os: indicates which OS the customer wants (feed to linux vm script) <br />
* ip(s): for linux customers, you should click “ipmap” and scroll down to the first available (green) ip amongst others on the same system. Click on the ip to copy it back to the “ip(s)” field on the form. For FreeBSD customers, the ips available for use are already assigned to the system and should be copied back to this screen once the jail is made (see below). For colo customers, choose an IP appropriate to the data center. i.e. for i2b pick an IP from the 229 block.<br />
* start date: should reflect the day the system was created. <br />
* asset tag: colo only<br />
* password: VPS only<br />
* cabinet: colo only<br />
* ats port: colo @ i2b only<br />
* monitored: should only be checked if the system’s ip/services were supplied to castle to place on the monitor (probes) list. <br />
<br />
NOTE: you must use a JavaScript enabled browser to enter new customers otherwise you won’t be given correct options for os templates.<br />
<br />
3. create the new system using either [[VPS_Management#jailmake|jailmake]] or [[VPS_Management#vm|vm]] scripts. For posterity, on our older linux systems and older OS's (pre virt17) we used to use a custom script for each OS version: vemakecentos3 vemakedebian40 vemakedebian30 vemakedebian31 vemakefedora2 vemakefedora6 vemakefedora7 vemakesuse100 vemakesuse93 vemakecentos4 vemakedebian31 vemakefedora4 vemakerh9 vemakeubuntu5 vemakefedora vemakefedora5 vemakesuse vemakeubuntu606 vemakeubuntu610 vemakeubuntu704<br />
<br />
[[VPS_Management#jailmake|jailmake]] and [[VPS_Management#vm|vm]] both email the new customer their welcome email.<br />
<br />
When you are done adding a customer, for both systems: copy back the dir and password (supplied by the VPS make script) into the form. For FreeBSD, copy back the IP which the make script will give back to you.<br />
<br />
When you are done filling out all the fields in the pending customer form, click “Activate”. This will create the customer in our database and remove them from the pending list. Nothing is emailed to the customer as a result of this action. If the customer paid via credit card, their info needs to be added manually.<br />
<br />
Discussion about choosing IPs ...<br />
<br />
When you make a new system, you have to choose what IP to give it. New FreeBSD servers are configured with a set amount of IPs, and as you add new systems to that freebsd server you can use one of the IPs assigned to the host, but not currently assigned to a VPS. You can use the js program to see what IPs are available for assigning to new customers. <br />
<br />
However, it is not that simple with the linux systems. The linux servers do not get the IPs of their customer systems bound to the actual machine. That is, even on a fully loaded linux system, if you run `ifconfig -a` from the base machine, you only see one IP - the main IP of that system. Further complicating matters is that linux systems can bind multiple IPs - therefore it is not possible to know that the next new linux system should just have the next IP as the last new one that was created.<br />
<br />
So, what we do is, new linux systems are simply assigned a starting IP, and no new machine (freebsd or linux) is assigned a base IP anywhere within 92-128 Ips of that IP. So you have 92-128 IPs to grow with for the linux systems that will live on that machine.<br />
<br />
So, you simply start creating systems from that starting point – and additional IPs that those systems need should also be added from that range, and even other linux systems on other virt machines should get new IPs from that block of IPs. To find the block of IPs in use, go thru bash history to find last ip used or assigned to a ve. Then, click the “ipmap” link (or pull up Mgmt. -> Reference -> IP Map). Ip’s not already assigned to a machine will show up as green. In general, we want to use ips which have never been assigned, or were relenquishsed years ago (or at least 30 days ago – IMPORTANT!!). <br />
D<br />
If a customer has selected an IP package – is paying for extra IPs now, then we’re pretty much obliged to assign all those IPs now. Otherwise, we only ever assign 1 IP.<br />
<br />
= The Welcome Emails =<br />
<br />
[[VPS_Management#jailmake|jailmake]] and [[VPS_Management#jailmake|vm]], as one of its arguments, asks for an email address. This email address is used to send a welcome email to the new user. However, the welcome email is not inside the jailmake/vemake script, and they are not on the jails themselves either. Here are all the various welcome emails we have:<br />
<br />
<pre>/usr/local/www/jc_pub/data/welcome-freebsdp (dynamic freebsd welcome email, 7.x 8.x)<br />
/usr/local/www/jc_pub/data/welcome-linux (dynamic linux welcome email)<br />
/usr/local/www/jc_pub/data/welcome-debian (dynamic debian/ubuntu welcome email)<br />
/usr/local/www/jc_pub/data/welcome-fedora (dynamic fedora/centos welcome email)<br />
/usr/local/www/jc_pub/data/welcome-freebsd (original, generic freebsd welcome email)<br />
/usr/local/www/jc_pub/data/welcome-freebsd6 (for freebsd6)<br />
/usr/local/www/jc_pub/data/welcome-freebsd7z (for freebsd7 with zfs DEPRECATED)</pre><br />
<br />
on the main johncompanies server.<br />
<br />
When the jailmake script is run, it issues the `fetch` (or wget) command to retrieve it, i.e.<br />
http://www.johncompanies.com/colocation/data/welcome-freebsdp<br />
<br />
and saves it as a temp file, mails it off.<br />
<br />
Welcome emails not noted as dynamic above, are emailed with the IP appended at the top to the email address specified. The password to these accounts was/is generic- not very good. Welcome emails noted as dynamic are parsed by the make script to include the IP and a generated, random root password. <br />
<br />
IMPORTANT NOTE: when creating an older-OS ve on a virt- this almost never happens (i.e. it’s created using the old-style vemakexxxx command), you must use the support address as the email address as the make script will be pulling down the dynamic welcome email when it expects to see the old, generic format and it won’t look good. When you receive the welcome email in support, format properly and resend to customer.<br />
<br />
This means three things:<br />
<br />
a) In order for jailmake to work, the johncompanies web server needs to be up and running, and that file needs to exist at that URL<br />
<br />
b) If you want to edit the welcome email, you only need to edit it on the web server, in one place<br />
<br />
c) If a customer for some reason does not receive the welcome email, then you need to go Mgmt -> Reference -> (file), copy and paste it into an email to them. For the old-generic emails, make sure to add this line to the very top:<br />
<br />
IP: (their IP)<br />
<br />
If the welcome email does not exist, jailmake will still work, in the sense that it will create the system, but no welcome email will be sent.<br />
<br />
= Dedicated Server Setup =<br />
<br />
Before starting the OS install you'll need to know the following pieces of info:<br />
<br />
*CID:<br />
*Customer has multiple colos: Y/N<br />
*Asset tag: JC-xxxx<br />
*rack/location: <br />
*Service/Package (and any deviations to B/W, nfs space, IPs, etc) and price:<br />
*RAM in system:<br />
*IPs included in plan:<br />
*OS (32 vs 64bit variant):<br />
*Hostname:<br />
*Disk partitions, including swap space:<br />
*Number of initial IPs to assign:<br />
*Timezone:<br />
<br />
Some to most of that info will be provided via the new signup page, assuming the customer ordered the server via our order page. The rest will comes from the sales/build department.<br />
<br />
<br />
Once the server is built, it should be installed in the rack, booted to the BIOS screen and labeled with the asset tag and the customer's CID (if available/established).<br />
<br />
If an [[IPKVM]] is not already attached, you will need to ascertain which one is available (look in ~user/kvm, usually open for editing in the p4 screen of the mailbox window).<br />
<br />
Before loading the server, there's a couple things to do in the BIOS screen:<br />
# make sure the date and time are set to UTC<br />
# make sure the server's power restore action is set to: last state (basically what we're trying to do here is if the server has power pulled, when power is restored it should turn back on- assuming it was on when power was pulled. This is what allows our ATS power cycling to work. If this is not set, when the ATS port is turned off and turned back on, the server may not turn back on.<br />
# set the boot order to disable network booting or other things that may slow down bootup. <br />
<br />
After saving and exiting the BIOS, if the server has a raid card installed, you will see the raid BIOS screen. You should enter this screen and setup a raid mirror, or whatever the customer has requested (if a special request was made sales will let you know). Usually we use a 3ware card, to enter the config screen, press ALT-3. Use spacebar to select the 2 drives, tab to 'create unit'. Use defaults. Create a raid1 mirror (unless customer asks for something else). Do not enable write cache, assuming no battery exists. F8 to save and exit.<br />
<br />
Reboot and load the OS- follow the instructions for pulling the ISO into the IPKVM and booting to it [[IPKVM|here]].<br />
<br />
The install should be for the server version of whatever OS was requested. We typically do not install anything other than an sshd (and ports (tree) for FreeBSD). We don't install a GUI environment. We do not setup auto updates. We do not encrypt home directories. <br />
<br />
When selecting and IP, take into consideration the data center: an install at i2b should use an IP from the IPs routed to i2b, and vice versa. A customer's package may come with multiple IPs, however unless they indicate they want more assigned initially (and show good reason for doing so) we only assign 1 IP. If they are to receive more IPs, you will/may need to configure those post-install (depending on the OS). Ideally all IPs they receive are on the same class C block.<br />
<br />
When picking a NIC (most of the installs we do are network versions and require the network to download OS components), it's somewhat of a crap shoot as to which NIC to choose and configure. Usually we try to begin pinging the IP we assign and watch for it to respond immediately after configuring the NIC. If it does not ping, you can:<br />
# ask the NOC to swap the network cable to the alternate port<br />
# go back and configure the alternate NIC (you may have to restart the OS install if it doesn't properly unconfigure the initial NIC, or you can't change it's config to alleviate an IP conflict)<br />
<br />
We set DNS: 69.55.229.3, 69.55.225.225 (for a server @ i2b), 69.55.225.225, 69.55.230.3 (for a server @ castle)<br />
<br />
We setup a regular user account: 'user' and we set that password to 'newrootNNNN' where NNNN is the sig digits of the customer's CID. i.e. for col01233 the password is 'newroot1233'. In FreeBSD, make sure to add 'user' to the wheel group (member of). <br />
If a root password is allowed to be set, we use the same password: 'newrootNNNN' <br />
<br />
Once the OS is installed and you've rebooted (after disconnecting the ISO from the KVM) you should confirm all is working: swap, ram, disk partitions, network is as it should be. If you were only able to assign 1 DNS server in setup, please add another to /etc/resolv.conf:<br />
<br />
@i2b: 69.55.229.3, 69.55.225.225<br />
@castle: 69.55.225.225, 69.55.230.3<br />
<br />
Assuming they have a 3ware raid card, you will need to transfer in the raid CLI tool and our handy script so the customer may check on the health of their raid array, as instructed/encouraged by the welcome email they will receive.<br />
<br />
This package is located on mail;<br />
<pre><br />
fetch http://johncompanies.com/tools/3ware/L64.tgz<br />
tar xvzf L64.tgz<br />
</pre><br />
<br />
The version you choose will of course depend on the OS/version you're installing. You should copy the tarball to the /usr/local/sbin dir (or something in their path) and untar'd. You should run the script included in the tarball called checkraid.sh. If it gives an error, this is due to the fact that on some servers the raid card is recognized as 'c1' and on others 'c0' or something different. Determine which it is for this server by running:<br />
tw_cli info<br />
<br />
which will give you output like:<br />
<pre>Ctl Model (V)Ports Drives Units NotOpt RRate VRate BBU<br />
------------------------------------------------------------------------<br />
c1 9650SE-8LPML 8 6 1 0 5 1 -<br />
</pre><br />
<br />
In this example it's 'c1' so edit checkraid.sh and change all instances of 'c0' to 'c1'<br />
<br />
If we're adding additional IPs (and were not able to do so during initial install), configure as follows:<br />
<br />
Centos/Fedora:<br />
<br />
<pre><br />
cd /etc/sysconfig/network-scripts/<br />
cp -p ifcfg-eth* ~<br />
fetch http://johncompanies.com/tools/network/centos/ifcfg-eth.tgz<br />
tar xvzf ifcfg-eth.tgz<br />
<br />
vi ifcfg-eth0:<br />
<br />
DEVICE=eth0<br />
TYPE=Ethernet<br />
ONBOOT=yes<br />
NM_CONTROLLED=yes<br />
BOOTPROTO=none<br />
IPADDR=69.55.231.XXX # change this to main IP address<br />
PREFIX=24<br />
GATEWAY=69.55.231.1 # change this to 229.1 if IP addr in other block<br />
DNS1=69.55.229.3<br />
DNS2=69.55.225.225<br />
DNS3=69.55.230.3<br />
DOMAIN="johncompanies.com"<br />
DEFROUTE=yes<br />
IPV4_FAILURE_FATAL=yes<br />
IPV6INIT=no<br />
NAME="System eth0"<br />
</pre><br />
<br />
<pre><br />
vi eth0:*<br />
<br />
DEVICE=eth0:0<br />
TYPE=Ethernet<br />
ONBOOT=yes<br />
NM_CONTROLLED=yes<br />
BOOTPROTO=none<br />
IPADDR=69.55.231.XXX # change this to IP address<br />
PREFIX=24<br />
IPV4_FAILURE_FATAL=yes<br />
IPV6INIT=no<br />
</pre><br />
<br />
Repeat above as needed for each IP address<br />
<br />
Remove the un-needed files.<br />
<pre><br />
rm ifcfg-eth0:[3-8] # change numbers to fit how many IP addresses are needed<br />
</pre><br />
<br />
Debian/Ubuntu:<br />
<br />
<pre><br />
cd /etc/network<br />
vi interfaces<br />
<br />
Old file was:<br />
# The loopback network interface<br />
auto lo<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
auto eth0<br />
iface eth0 inet static<br />
address 69.55.227.4<br />
netmask 255.255.255.0<br />
network 69.55.227.0<br />
broadcast 69.55.227.255<br />
gateway 69.55.227.1<br />
# dns-* options are implemented by the resolvconf package, if installed<br />
dns-nameservers 69.55.225.225<br />
dns-search yeppernet.com<br />
<br />
new file is:<br />
# The loopback network interface<br />
auto lo<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
auto eth0<br />
iface eth0 inet static<br />
address 69.55.227.44<br />
netmask 255.255.255.0<br />
network 69.55.227.0<br />
broadcast 69.55.227.255<br />
gateway 69.55.227.1<br />
# dns-* options are implemented by the resolvconf package, if installed<br />
dns-nameservers 69.55D.225.225<br />
dns-search yeppernet.com<br />
<br />
auto eth0:0<br />
iface eth0:0 inet static<br />
address 69.55.227.55<br />
netmask 255.255.255.0<br />
network 69.55.227.0<br />
broadcast 69.55.227.255<br />
gateway 69.55.227.1<br />
<br />
</pre><br />
/etc/init.d/networking restart<br />
(make sure you have console in case you screw that up)<br />
<br />
<pre>ifconfig<br />
eth0 Link encap:Ethernet HWaddr 00:30:48:28:9d:50<br />
inet addr:69.55.227.44 Bcast:69.55.227.255 Mask:255.255.255.0<br />
inet6 addr: fe80::230:48ff:fe28:9d50/64 Scope:Link<br />
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br />
RX packets:2850149 errors:0 dropped:0 overruns:0 frame:0<br />
TX packets:2595566 errors:0 dropped:0 overruns:0 carrier:0<br />
collisions:0 txqueuelen:100<br />
RX bytes:1193068127 (1.1 GB) TX bytes:2046984105 (1.9 GB)<br />
Base address:0x3000 Memory:fc400000-fc420000<br />
<br />
eth0:0 Link encap:Ethernet HWaddr 00:30:48:28:9d:50<br />
inet addr:69.55.227.55 Bcast:69.55.227.255 Mask:255.255.255.0<br />
inet6 addr: fe80::230:48ff:fe28:9d50/64 Scope:Link<br />
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br />
RX packets:2850149 errors:0 dropped:0 overruns:0 frame:0<br />
TX packets:2595566 errors:0 dropped:0 overruns:0 carrier:0<br />
collisions:0 txqueuelen:100<br />
RX bytes:1193068127 (1.1 GB) TX bytes:2046984105 (1.9 GB)<br />
Base address:0x3000 Memory:fc400000-fc420000<br />
</pre><br />
<br />
Ubuntu 18.04<br />
<br />
Ubuntu 18.04 uses netplan instead of /etc/network/interfaces.<br />
<br />
<pre><br />
vi /etc/neplan/50-cloud-init.yaml<br />
<br />
# This file is generated from information provided by<br />
# the datasource. Changes to it will not persist across an instance.<br />
# To disable cloud-init's network configuration capabilities, write a file<br />
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:<br />
# network: {config: disabled}<br />
network:<br />
version: 2<br />
renderer: networkd<br />
ethernets:<br />
enp0s25:<br />
dhcp4: no<br />
addresses: [69.55.229.26/24]<br />
gateway4: 69.55.229.1 <br />
nameservers:<br />
addresses: [69.55.229.3,8.8.8.8,8.8.4.4]<br />
<br />
netplan apply<br />
</pre><br />
<br />
The very last thing you should do before logging off the server is to run:<br />
history -c<br />
in every shell you were in so they don't see what we were doing :)<br />
<br />
Assuming this server is connected to an ATS, you'll want to confirm the server is connected to the port you think it is, and test the functionality- that you can power cycle the port and the server will come back up when power is restored. You should do a test power cycle (via the mgmt or AM-based ATS control) while the server is in post or BIOS. Basically, just NOT while the OS is running and disks mounted.<br />
<br />
You should confirm the switch ports to which the server is connected (maybe has a 2nd port for the RMM). While the server is rebooting during your ATS test, you can observe output on the switch console that looks like:<br />
<pre>.Mar 7 09:47:45 PST: %LINK-3-UPDOWN: Interface FastEthernet0/14, changed state to down<br />
.Mar 7 09:47:46 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/14, changed state to down<br />
.Mar 7 09:47:47 PST: %LINK-3-UPDOWN: Interface FastEthernet0/14, changed state to up<br />
.Mar 7 09:47:48 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/14, changed state to up<br />
</pre><br />
This confirms the server is on port 14.<br />
<br />
Take this info (ATS, switch ports plus the asset tag) and:<br />
# activate (or add) the server. Update the cabinet, ATS and port, asset tag. (Switch port not kept in mgmt)<br />
# update the [[cabinetmap|cabinet map]] to add the server to the correct location, along with the ATS port, and switch port(s)<br />
# login to the switch serial console and add the customer's CID (and hostname if they have mult. servers) to the port(s) description tag. Repeat for the RMM port if they have one:<br />
<pre>switch-p25#conf t<br />
Enter configuration commands, one per line. End with CNTL/Z.<br />
switch-p25(config)#int fa0/10<br />
switch-p25(config-if)#des col01233 myhostname<br />
switch-p25(config)#int fa0/11<br />
switch-p25(config-if)#des col01233 myhostname (RMM)<br />
switch-p25(config-if)#end<br />
switch-p25#<br />
.Mar 7 11:37:17 PST: %SYS-5-CONFIG_I: Configured from console by console<br />
switch-p25#wr me<br />
Building configuration...<br />
[OK]<br />
switch-p25#<br />
</pre><br />
<br />
At this point it's safe to hand the server over to the customer. Use the 'new colo welcome' paste. Edit for:<br />
# IP(s)<br />
# if it's FreeBSD use the 'root' & 'user' password line, if it's Ubuntu use/edit the 'root' password line, depending on if you set the root password or not, or what you setup for the normal user account<br />
# the correct DNS depending on which data center they're at (remove ns3c if the server is at castle)<br />
# if they have no raid card, remove that section<br />
# if they have an RMM, include the RMM paste somewhere in the email<br />
<br />
If they're already a customer, you can exclude the info about their AM login info (since they have it already). If they're not, you'll need to reset their AM pass to see it.<br />
<br />
Lastly, assuming this server was installed for a new customer and the customer was not yet activated, the CID was probably not established and the server has no CID label. You'll need to make a note to label it (and add the hostname too if they have multiple servers) next time you're at the data center, or ask the noc staff to do it for you.<br />
<br />
== Dedicated Server Install Checklist ==<br />
<br />
o Install OS<br />
<br />
o Add JCI - adduser, wheel, visudo<br />
<pre><br />
adduser -u 955 jci<br />
passwd jci<br />
''l4mph0st''<br />
adduser -u 1000 col0XXX<br />
passwd col0XXXX <br />
''col0XXXX'' (XXXX is their col0XXXX number)<br />
vi /etc/group<br />
( add "jci,user" to group wheel and/or sudo )<br />
visudo<br />
( enable group wheel to do anything with password )<br />
</pre><br />
<br />
o Install openssh-clients (yum install openssh-clients)<br />
<pre><br />
yum -y install openssh-clients<br />
</pre><br />
<br />
o Update OS<br />
<pre><br />
yum -y update (for CentOS)<br />
</pre><br />
<br />
o checkraid.sh<br />
As noted above install our RAID disk checks<br />
<br />
o config IP addresses<br />
<br />
o clear last logins<br />
<pre><br />
cp /dev/null /var/log/wtmp<br />
</pre><br />
<br />
o clear history<br />
<pre><br />
cp /dev/null ~/.bash_history<br />
history -c<br />
shutdown -h now<br />
</pre><br />
<br />
o For a suspected spammer, add rule to firewall2 to limit output for mail ports (ie.)<br />
before rule 50<br />
<pre><br />
ipfw add 37 pipe 8 ip from 69.55.231.104/30 to any dst-port 25,465,587<br />
ipfw add 37 pipe 8 ip from 69.55.231.108 to any dst-port 25,465,587<br />
</pre><br />
<br />
=== After installed in rack ===<br />
<br />
check ssh<br />
label switch port<br />
update cabinet map<br />
update mgmt cabinet number<br />
<br />
== RMM Setup ==<br />
<br />
The RMM must be configured via the server's BIOS screen:<br />
<br />
Server management > BMC LAN config:<BR><br />
down to Intel RMM3 LAN config: <BR><br />
IP (static) , mask, g/w<BR><br />
<br />
user config: <BR><br />
disable anonymous. <BR><br />
root: set pass to newrootNNNN</div>75.80.128.147https://wiki.jcihosting.com/index.php?title=New_Signups&diff=2219New Signups2020-03-29T18:55:44Z<p>75.80.128.147: /* Dedicated Server Setup */</p>
<hr />
<div>__TOC__<br />
TODO- review<br />
<br />
= Open source discount signups =<br />
<br />
URLS to give to customers:<br />
https://secure.johncompanies.com/signup/step1.html?svc=bsd-oss<br />
<br />
https://secure.johncompanies.com/signup/step1.html?svc=linux-oss<br />
<br />
= New Signups = <br />
New customers sign up for service on our web based forms at<br />
www.johncompanies.com <br />
<br />
When a signup occurs an email is sent to support@johncompanies.com <br />
and signups get written to two files:<br />
<br />
/usr/local/www/jc_pub/data/pending<br />
<br />
and<br />
<br />
/usr/local/www/jc_pub/data/log/log.pending<br />
<br />
The second file is simply a running log of signups, and should not be used for anything. The only time it is handy is if you are editing the pending file, then save it, and are told that the file has changed since you started editing it. This means that someone signed up while you were editing the pending file. What you should do is force the save (thus losing their signup) and then cut and paste the lost signup from /htdocs/colocation/data/log/log.pending.<br />
<br />
Lines in the pending file look like this:<br />
<br />
<pre>2005-11-08;newcastle;Frederick Wilson;;1405 South Adams;Fort Worth;Texas;76104;US;referred by nmrc.org;LM-1;2;Limit 10;1;207.13.31.48;106<br />
vaifan@airmail.net;817-798-8637;on;on;<br />
new;add2;;;replace;replace;preserve;elaine.commadev.com</pre><br />
<br />
The fields are defined as follows:<br />
<br />
<pre>signup date;hostname;last, first;company; address;city;state;zip;country;referred by;package id;bandwidth overage option;if gobut is selected, $ amount to limit is listed here;payment method;source ip;template id;<br />
email;phone #;admin;billing;alt<br />
new/replace system;create new account/add to existing;hostname of server being replaced;new IP pref;update/merge contacts;update/merge owner info;update/preserve traffic overage;</pre><br />
<br />
When a customer signs up an email will be sent to support@johncompanies.com containing the log entry in the format above<br />
<br />
The process for setting up a new customer is as follows:<br />
<br />
1. Go to the pending customers screen (choose Mgmt site -> New Signups). New customers will be at the bottom of the "VPS Signups" and "Colo Signups" sections. If they’ve paid automatically (via PayPal) the word “PAID” will appear in the “pmt status” column. The other thing you’ll see in this column is “paid- not cleared”- this means they paid with eCheck (via PayPal) and the check has not cleared. This normally takes 3 biz. days and we wait till it clears and shows “PAID” before setting up the server. Customers who pay with a credit card will not appear as "PAID" here. In that case, an email will be sent to support@johncompanies.com indicating a payment has been made (or failed). Once the payment has been paid/cleared, click “process….” <br />
<br />
Before setting someone up, you need to decide whether the signup is fraudulent. Examples of fraud can be found in /usr/local/www/jc_pub/data/fraud-examples <br />
Generally, <br />
* NOT FRAUD = non-anonymous email accounts that match who they are (dan@brockman.com, where the person is Dan Brockman)<br />
* NOT FRAUD = referred by a customer that actually exists or by kuro5hin<br />
* MAYBE FRAUD = referred by ‘web search’ or ‘google’<br />
* MAYBE FRAUD = generic sounding address<br />
* MAYBE FRAUD = IP address is out of country and address is in US (use dnsstuff link on main signups screen to lookup where IP is)<br />
* FRAUD = they don’t type in: referred by and/or hostname and/or give the default answer for bandwidth is stop<br />
* FRAUD = multiple orders each from the same IP and/or using same email address<br />
* FRAUD = no hostname or hacker-sounding hostname: 3v1l0n3 (evilone)<br />
<br />
If you’re ever in doubt, call the card owner using the phone number given at signup or call the bank number provided for the card. Never use the email as it’s likely the thief’s and not the cardholder’s.<br />
<br />
Almost all the info needed is pre-entered into this screen, but some fields will need attention:<br />
<br />
* System: The system is already selected based on the template/OS the customer selected. <br />
* Directory: information provided for you to enter once the system is created. (leave bank for managedcolo)<br />
* Disk: indicates how much disk space the server should have. (leave bank for managedcolo)<br />
* Hostname: self-explanatory. <br />
* veid: (linux systems only) should be filled in with the significant digits of the customer ID, ex: col01340 = veid 1340. <br />
* os: indicates which OS the customer wants (feed to linux vm script) <br />
* ip(s): for linux customers, you should click “ipmap” and scroll down to the first available (green) ip amongst others on the same system. Click on the ip to copy it back to the “ip(s)” field on the form. For FreeBSD customers, the ips available for use are already assigned to the system and should be copied back to this screen once the jail is made (see below). For colo customers, choose an IP appropriate to the data center. i.e. for i2b pick an IP from the 229 block.<br />
* start date: should reflect the day the system was created. <br />
* asset tag: colo only<br />
* password: VPS only<br />
* cabinet: colo only<br />
* ats port: colo @ i2b only<br />
* monitored: should only be checked if the system’s ip/services were supplied to castle to place on the monitor (probes) list. <br />
<br />
NOTE: you must use a JavaScript enabled browser to enter new customers otherwise you won’t be given correct options for os templates.<br />
<br />
3. create the new system using either [[VPS_Management#jailmake|jailmake]] or [[VPS_Management#vm|vm]] scripts. For posterity, on our older linux systems and older OS's (pre virt17) we used to use a custom script for each OS version: vemakecentos3 vemakedebian40 vemakedebian30 vemakedebian31 vemakefedora2 vemakefedora6 vemakefedora7 vemakesuse100 vemakesuse93 vemakecentos4 vemakedebian31 vemakefedora4 vemakerh9 vemakeubuntu5 vemakefedora vemakefedora5 vemakesuse vemakeubuntu606 vemakeubuntu610 vemakeubuntu704<br />
<br />
[[VPS_Management#jailmake|jailmake]] and [[VPS_Management#vm|vm]] both email the new customer their welcome email.<br />
<br />
When you are done adding a customer, for both systems: copy back the dir and password (supplied by the VPS make script) into the form. For FreeBSD, copy back the IP which the make script will give back to you.<br />
<br />
When you are done filling out all the fields in the pending customer form, click “Activate”. This will create the customer in our database and remove them from the pending list. Nothing is emailed to the customer as a result of this action. If the customer paid via credit card, their info needs to be added manually.<br />
<br />
Discussion about choosing IPs ...<br />
<br />
When you make a new system, you have to choose what IP to give it. New FreeBSD servers are configured with a set amount of IPs, and as you add new systems to that freebsd server you can use one of the IPs assigned to the host, but not currently assigned to a VPS. You can use the js program to see what IPs are available for assigning to new customers. <br />
<br />
However, it is not that simple with the linux systems. The linux servers do not get the IPs of their customer systems bound to the actual machine. That is, even on a fully loaded linux system, if you run `ifconfig -a` from the base machine, you only see one IP - the main IP of that system. Further complicating matters is that linux systems can bind multiple IPs - therefore it is not possible to know that the next new linux system should just have the next IP as the last new one that was created.<br />
<br />
So, what we do is, new linux systems are simply assigned a starting IP, and no new machine (freebsd or linux) is assigned a base IP anywhere within 92-128 Ips of that IP. So you have 92-128 IPs to grow with for the linux systems that will live on that machine.<br />
<br />
So, you simply start creating systems from that starting point – and additional IPs that those systems need should also be added from that range, and even other linux systems on other virt machines should get new IPs from that block of IPs. To find the block of IPs in use, go thru bash history to find last ip used or assigned to a ve. Then, click the “ipmap” link (or pull up Mgmt. -> Reference -> IP Map). Ip’s not already assigned to a machine will show up as green. In general, we want to use ips which have never been assigned, or were relenquishsed years ago (or at least 30 days ago – IMPORTANT!!). <br />
D<br />
If a customer has selected an IP package – is paying for extra IPs now, then we’re pretty much obliged to assign all those IPs now. Otherwise, we only ever assign 1 IP.<br />
<br />
= The Welcome Emails =<br />
<br />
[[VPS_Management#jailmake|jailmake]] and [[VPS_Management#jailmake|vm]], as one of its arguments, asks for an email address. This email address is used to send a welcome email to the new user. However, the welcome email is not inside the jailmake/vemake script, and they are not on the jails themselves either. Here are all the various welcome emails we have:<br />
<br />
<pre>/usr/local/www/jc_pub/data/welcome-freebsdp (dynamic freebsd welcome email, 7.x 8.x)<br />
/usr/local/www/jc_pub/data/welcome-linux (dynamic linux welcome email)<br />
/usr/local/www/jc_pub/data/welcome-debian (dynamic debian/ubuntu welcome email)<br />
/usr/local/www/jc_pub/data/welcome-fedora (dynamic fedora/centos welcome email)<br />
/usr/local/www/jc_pub/data/welcome-freebsd (original, generic freebsd welcome email)<br />
/usr/local/www/jc_pub/data/welcome-freebsd6 (for freebsd6)<br />
/usr/local/www/jc_pub/data/welcome-freebsd7z (for freebsd7 with zfs DEPRECATED)</pre><br />
<br />
on the main johncompanies server.<br />
<br />
When the jailmake script is run, it issues the `fetch` (or wget) command to retrieve it, i.e.<br />
http://www.johncompanies.com/colocation/data/welcome-freebsdp<br />
<br />
and saves it as a temp file, mails it off.<br />
<br />
Welcome emails not noted as dynamic above, are emailed with the IP appended at the top to the email address specified. The password to these accounts was/is generic- not very good. Welcome emails noted as dynamic are parsed by the make script to include the IP and a generated, random root password. <br />
<br />
IMPORTANT NOTE: when creating an older-OS ve on a virt- this almost never happens (i.e. it’s created using the old-style vemakexxxx command), you must use the support address as the email address as the make script will be pulling down the dynamic welcome email when it expects to see the old, generic format and it won’t look good. When you receive the welcome email in support, format properly and resend to customer.<br />
<br />
This means three things:<br />
<br />
a) In order for jailmake to work, the johncompanies web server needs to be up and running, and that file needs to exist at that URL<br />
<br />
b) If you want to edit the welcome email, you only need to edit it on the web server, in one place<br />
<br />
c) If a customer for some reason does not receive the welcome email, then you need to go Mgmt -> Reference -> (file), copy and paste it into an email to them. For the old-generic emails, make sure to add this line to the very top:<br />
<br />
IP: (their IP)<br />
<br />
If the welcome email does not exist, jailmake will still work, in the sense that it will create the system, but no welcome email will be sent.<br />
<br />
= Dedicated Server Setup =<br />
<br />
Before starting the OS install you'll need to know the following pieces of info:<br />
<br />
*CID:<br />
*Customer has multiple colos: Y/N<br />
*Asset tag: JC-xxxx<br />
*rack/location: <br />
*Service/Package (and any deviations to B/W, nfs space, IPs, etc) and price:<br />
*RAM in system:<br />
*IPs included in plan:<br />
*OS (32 vs 64bit variant):<br />
*Hostname:<br />
*Disk partitions, including swap space:<br />
*Number of initial IPs to assign:<br />
*Timezone:<br />
<br />
Some to most of that info will be provided via the new signup page, assuming the customer ordered the server via our order page. The rest will comes from the sales/build department.<br />
<br />
<br />
Once the server is built, it should be installed in the rack, booted to the BIOS screen and labeled with the asset tag and the customer's CID (if available/established).<br />
<br />
If an [[IPKVM]] is not already attached, you will need to ascertain which one is available (look in ~user/kvm, usually open for editing in the p4 screen of the mailbox window).<br />
<br />
Before loading the server, there's a couple things to do in the BIOS screen:<br />
# make sure the date and time are set to UTC<br />
# make sure the server's power restore action is set to: last state (basically what we're trying to do here is if the server has power pulled, when power is restored it should turn back on- assuming it was on when power was pulled. This is what allows our ATS power cycling to work. If this is not set, when the ATS port is turned off and turned back on, the server may not turn back on.<br />
# set the boot order to disable network booting or other things that may slow down bootup. <br />
<br />
After saving and exiting the BIOS, if the server has a raid card installed, you will see the raid BIOS screen. You should enter this screen and setup a raid mirror, or whatever the customer has requested (if a special request was made sales will let you know). Usually we use a 3ware card, to enter the config screen, press ALT-3. Use spacebar to select the 2 drives, tab to 'create unit'. Use defaults. Create a raid1 mirror (unless customer asks for something else). Do not enable write cache, assuming no battery exists. F8 to save and exit.<br />
<br />
Reboot and load the OS- follow the instructions for pulling the ISO into the IPKVM and booting to it [[IPKVM|here]].<br />
<br />
The install should be for the server version of whatever OS was requested. We typically do not install anything other than an sshd (and ports (tree) for FreeBSD). We don't install a GUI environment. We do not setup auto updates. We do not encrypt home directories. <br />
<br />
When selecting and IP, take into consideration the data center: an install at i2b should use an IP from the IPs routed to i2b, and vice versa. A customer's package may come with multiple IPs, however unless they indicate they want more assigned initially (and show good reason for doing so) we only assign 1 IP. If they are to receive more IPs, you will/may need to configure those post-install (depending on the OS). Ideally all IPs they receive are on the same class C block.<br />
<br />
When picking a NIC (most of the installs we do are network versions and require the network to download OS components), it's somewhat of a crap shoot as to which NIC to choose and configure. Usually we try to begin pinging the IP we assign and watch for it to respond immediately after configuring the NIC. If it does not ping, you can:<br />
# ask the NOC to swap the network cable to the alternate port<br />
# go back and configure the alternate NIC (you may have to restart the OS install if it doesn't properly unconfigure the initial NIC, or you can't change it's config to alleviate an IP conflict)<br />
<br />
We set DNS: 69.55.229.3, 69.55.225.225 (for a server @ i2b), 69.55.225.225, 69.55.230.3 (for a server @ castle)<br />
<br />
We setup a regular user account: 'user' and we set that password to 'newrootNNNN' where NNNN is the sig digits of the customer's CID. i.e. for col01233 the password is 'newroot1233'. In FreeBSD, make sure to add 'user' to the wheel group (member of). <br />
If a root password is allowed to be set, we use the same password: 'newrootNNNN' <br />
<br />
Once the OS is installed and you've rebooted (after disconnecting the ISO from the KVM) you should confirm all is working: swap, ram, disk partitions, network is as it should be. If you were only able to assign 1 DNS server in setup, please add another to /etc/resolv.conf:<br />
<br />
@i2b: 69.55.229.3, 69.55.225.225<br />
@castle: 69.55.225.225, 69.55.230.3<br />
<br />
Assuming they have a 3ware raid card, you will need to transfer in the raid CLI tool and our handy script so the customer may check on the health of their raid array, as instructed/encouraged by the welcome email they will receive.<br />
<br />
This package is located on mail;<br />
<pre><br />
fetch http://johncompanies.com/tools/3ware/L64.tgz<br />
tar xvzf L64.tgz<br />
</pre><br />
<br />
The version you choose will of course depend on the OS/version you're installing. You should copy the tarball to the /usr/local/sbin dir (or something in their path) and untar'd. You should run the script included in the tarball called checkraid.sh. If it gives an error, this is due to the fact that on some servers the raid card is recognized as 'c1' and on others 'c0' or something different. Determine which it is for this server by running:<br />
tw_cli info<br />
<br />
which will give you output like:<br />
<pre>Ctl Model (V)Ports Drives Units NotOpt RRate VRate BBU<br />
------------------------------------------------------------------------<br />
c1 9650SE-8LPML 8 6 1 0 5 1 -<br />
</pre><br />
<br />
In this example it's 'c1' so edit checkraid.sh and change all instances of 'c0' to 'c1'<br />
<br />
If we're adding additional IPs (and were not able to do so during initial install), configure as follows:<br />
<br />
Centos/Fedora:<br />
<br />
<pre><br />
cd /etc/sysconfig/network-scripts/<br />
cp -p ifcfg-eth* ~<br />
fetch http://johncompanies.com/tools/network/centos/ifcfg-eth.tgz<br />
tar xvzf ifcfg-eth.tgz<br />
<br />
vi ifcfg-eth0:<br />
<br />
DEVICE=eth0<br />
TYPE=Ethernet<br />
ONBOOT=yes<br />
NM_CONTROLLED=yes<br />
BOOTPROTO=none<br />
IPADDR=69.55.231.XXX # change this to main IP address<br />
PREFIX=24<br />
GATEWAY=69.55.231.1 # change this to 229.1 if IP addr in other block<br />
DNS1=69.55.229.3<br />
DNS2=69.55.225.225<br />
DNS3=69.55.230.3<br />
DOMAIN="johncompanies.com"<br />
DEFROUTE=yes<br />
IPV4_FAILURE_FATAL=yes<br />
IPV6INIT=no<br />
NAME="System eth0"<br />
</pre><br />
<br />
<pre><br />
vi eth0:*<br />
<br />
DEVICE=eth0:0<br />
TYPE=Ethernet<br />
ONBOOT=yes<br />
NM_CONTROLLED=yes<br />
BOOTPROTO=none<br />
IPADDR=69.55.231.XXX # change this to IP address<br />
PREFIX=24<br />
IPV4_FAILURE_FATAL=yes<br />
IPV6INIT=no<br />
</pre><br />
<br />
Repeat above as needed for each IP address<br />
<br />
Remove the un-needed files.<br />
<pre><br />
rm ifcfg-eth0:[3-8] # change numbers to fit how many IP addresses are needed<br />
</pre><br />
<br />
Debian/Ubuntu:<br />
<br />
<pre><br />
cd /etc/network<br />
vi interfaces<br />
<br />
Old file was:<br />
# The loopback network interface<br />
auto lo<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
auto eth0<br />
iface eth0 inet static<br />
address 69.55.227.4<br />
netmask 255.255.255.0<br />
network 69.55.227.0<br />
broadcast 69.55.227.255<br />
gateway 69.55.227.1<br />
# dns-* options are implemented by the resolvconf package, if installed<br />
dns-nameservers 69.55.225.225<br />
dns-search yeppernet.com<br />
<br />
new file is:<br />
# The loopback network interface<br />
auto lo<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
auto eth0<br />
iface eth0 inet static<br />
address 69.55.227.44<br />
netmask 255.255.255.0<br />
network 69.55.227.0<br />
broadcast 69.55.227.255<br />
gateway 69.55.227.1<br />
# dns-* options are implemented by the resolvconf package, if installed<br />
dns-nameservers 69.55D.225.225<br />
dns-search yeppernet.com<br />
<br />
auto eth0:0<br />
iface eth0:0 inet static<br />
address 69.55.227.55<br />
netmask 255.255.255.0<br />
network 69.55.227.0<br />
broadcast 69.55.227.255<br />
gateway 69.55.227.1<br />
<br />
</pre><br />
/etc/init.d/networking restart<br />
(make sure you have console in case you screw that up)<br />
<br />
<pre>ifconfig<br />
eth0 Link encap:Ethernet HWaddr 00:30:48:28:9d:50<br />
inet addr:69.55.227.44 Bcast:69.55.227.255 Mask:255.255.255.0<br />
inet6 addr: fe80::230:48ff:fe28:9d50/64 Scope:Link<br />
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br />
RX packets:2850149 errors:0 dropped:0 overruns:0 frame:0<br />
TX packets:2595566 errors:0 dropped:0 overruns:0 carrier:0<br />
collisions:0 txqueuelen:100<br />
RX bytes:1193068127 (1.1 GB) TX bytes:2046984105 (1.9 GB)<br />
Base address:0x3000 Memory:fc400000-fc420000<br />
<br />
eth0:0 Link encap:Ethernet HWaddr 00:30:48:28:9d:50<br />
inet addr:69.55.227.55 Bcast:69.55.227.255 Mask:255.255.255.0<br />
inet6 addr: fe80::230:48ff:fe28:9d50/64 Scope:Link<br />
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br />
RX packets:2850149 errors:0 dropped:0 overruns:0 frame:0<br />
TX packets:2595566 errors:0 dropped:0 overruns:0 carrier:0<br />
collisions:0 txqueuelen:100<br />
RX bytes:1193068127 (1.1 GB) TX bytes:2046984105 (1.9 GB)<br />
Base address:0x3000 Memory:fc400000-fc420000<br />
</pre><br />
<br />
Ubuntu 18.04<br />
<br />
Ubuntu 18.04 uses netplan instead of /etc/network/interfaces.<br />
<br />
<pre><br />
vi /etc/neplan/50-cloud-init.yaml<br />
<br />
# This file is generated from information provided by<br />
# the datasource. Changes to it will not persist across an instance.<br />
# To disable cloud-init's network configuration capabilities, write a file<br />
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:<br />
# network: {config: disabled}<br />
network:<br />
version: 2<br />
renderer: networkd<br />
ethernets:<br />
enp0s25:<br />
dhcp4: no<br />
addresses: [69.55.229.26/24]<br />
gateway4: 69.55.229.1 <br />
nameservers:<br />
addresses: [69.55.229.3,8.8.8.8,8.8.4.4]<br />
<br />
netplan apply<br />
</pre><br />
<br />
The very last thing you should do before logging off the server is to run:<br />
history -c<br />
in every shell you were in so they don't see what we were doing :)<br />
<br />
Assuming this server is connected to an ATS, you'll want to confirm the server is connected to the port you think it is, and test the functionality- that you can power cycle the port and the server will come back up when power is restored. You should do a test power cycle (via the mgmt or AM-based ATS control) while the server is in post or BIOS. Basically, just NOT while the OS is running and disks mounted.<br />
<br />
You should confirm the switch ports to which the server is connected (maybe has a 2nd port for the RMM). While the server is rebooting during your ATS test, you can observe output on the switch console that looks like:<br />
<pre>.Mar 7 09:47:45 PST: %LINK-3-UPDOWN: Interface FastEthernet0/14, changed state to down<br />
.Mar 7 09:47:46 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/14, changed state to down<br />
.Mar 7 09:47:47 PST: %LINK-3-UPDOWN: Interface FastEthernet0/14, changed state to up<br />
.Mar 7 09:47:48 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/14, changed state to up<br />
</pre><br />
This confirms the server is on port 14.<br />
<br />
Take this info (ATS, switch ports plus the asset tag) and:<br />
# activate (or add) the server. Update the cabinet, ATS and port, asset tag. (Switch port not kept in mgmt)<br />
# update the [[cabinetmap|cabinet map]] to add the server to the correct location, along with the ATS port, and switch port(s)<br />
# login to the switch serial console and add the customer's CID (and hostname if they have mult. servers) to the port(s) description tag. Repeat for the RMM port if they have one:<br />
<pre>switch-p25#conf t<br />
Enter configuration commands, one per line. End with CNTL/Z.<br />
switch-p25(config)#int fa0/10<br />
switch-p25(config-if)#des col01233 myhostname<br />
switch-p25(config)#int fa0/11<br />
switch-p25(config-if)#des col01233 myhostname (RMM)<br />
switch-p25(config-if)#end<br />
switch-p25#<br />
.Mar 7 11:37:17 PST: %SYS-5-CONFIG_I: Configured from console by console<br />
switch-p25#wr me<br />
Building configuration...<br />
[OK]<br />
switch-p25#<br />
</pre><br />
<br />
At this point it's safe to hand the server over to the customer. Use the 'new colo welcome' paste. Edit for:<br />
# IP(s)<br />
# if it's FreeBSD use the 'root' & 'user' password line, if it's Ubuntu use/edit the 'root' password line, depending on if you set the root password or not, or what you setup for the normal user account<br />
# the correct DNS depending on which data center they're at (remove ns3c if the server is at castle)<br />
# if they have no raid card, remove that section<br />
# if they have an RMM, include the RMM paste somewhere in the email<br />
<br />
If they're already a customer, you can exclude the info about their AM login info (since they have it already). If they're not, you'll need to reset their AM pass to see it.<br />
<br />
Lastly, assuming this server was installed for a new customer and the customer was not yet activated, the CID was probably not established and the server has no CID label. You'll need to make a note to label it (and add the hostname too if they have multiple servers) next time you're at the data center, or ask the noc staff to do it for you.<br />
<br />
== Dedicated Server Install Checklist ==<br />
<br />
o Install OS<br />
<br />
o Add JCI - adduser, wheel, visudo<br />
<pre><br />
adduser -u 955 jci<br />
passwd jci<br />
''l4mph0st''<br />
adduser -u 1000 user<br />
passwd user <br />
''newrootXXXX'' (XXXX is their col0XXXX number)<br />
vi /etc/group<br />
( add "jci,user" to group wheel and/or sudo )<br />
visudo<br />
( enable group wheel to do anything with password )<br />
</pre><br />
<br />
o Install openssh-clients (yum install openssh-clients)<br />
<pre><br />
yum -y install openssh-clients<br />
</pre><br />
<br />
o Update OS<br />
<pre><br />
yum -y update (for CentOS)<br />
</pre><br />
<br />
o checkraid.sh<br />
As noted above install our RAID disk checks<br />
<br />
o config IP addresses<br />
<br />
o clear last logins<br />
<pre><br />
cp /dev/null /var/log/wtmp<br />
</pre><br />
<br />
o clear history<br />
<pre><br />
cp /dev/null ~/.bash_history<br />
history -c<br />
shutdown -h now<br />
</pre><br />
<br />
o For a suspected spammer, add rule to firewall2 to limit output for mail ports (ie.)<br />
before rule 50<br />
<pre><br />
ipfw add 37 pipe 8 ip from 69.55.231.104/30 to any dst-port 25,465,587<br />
ipfw add 37 pipe 8 ip from 69.55.231.108 to any dst-port 25,465,587<br />
</pre><br />
<br />
=== After installed in rack ===<br />
<br />
check ssh<br />
label switch port<br />
update cabinet map<br />
update mgmt cabinet number<br />
<br />
== RMM Setup ==<br />
<br />
The RMM must be configured via the server's BIOS screen:<br />
<br />
Server management > BMC LAN config:<BR><br />
down to Intel RMM3 LAN config: <BR><br />
IP (static) , mask, g/w<BR><br />
<br />
user config: <BR><br />
disable anonymous. <BR><br />
root: set pass to newrootNNNN</div>75.80.128.147https://wiki.jcihosting.com/index.php?title=New_Signups&diff=2218New Signups2020-03-29T18:52:50Z<p>75.80.128.147: /* Dedicated Server Setup */</p>
<hr />
<div>__TOC__<br />
TODO- review<br />
<br />
= Open source discount signups =<br />
<br />
URLS to give to customers:<br />
https://secure.johncompanies.com/signup/step1.html?svc=bsd-oss<br />
<br />
https://secure.johncompanies.com/signup/step1.html?svc=linux-oss<br />
<br />
= New Signups = <br />
New customers sign up for service on our web based forms at<br />
www.johncompanies.com <br />
<br />
When a signup occurs an email is sent to support@johncompanies.com <br />
and signups get written to two files:<br />
<br />
/usr/local/www/jc_pub/data/pending<br />
<br />
and<br />
<br />
/usr/local/www/jc_pub/data/log/log.pending<br />
<br />
The second file is simply a running log of signups, and should not be used for anything. The only time it is handy is if you are editing the pending file, then save it, and are told that the file has changed since you started editing it. This means that someone signed up while you were editing the pending file. What you should do is force the save (thus losing their signup) and then cut and paste the lost signup from /htdocs/colocation/data/log/log.pending.<br />
<br />
Lines in the pending file look like this:<br />
<br />
<pre>2005-11-08;newcastle;Frederick Wilson;;1405 South Adams;Fort Worth;Texas;76104;US;referred by nmrc.org;LM-1;2;Limit 10;1;207.13.31.48;106<br />
vaifan@airmail.net;817-798-8637;on;on;<br />
new;add2;;;replace;replace;preserve;elaine.commadev.com</pre><br />
<br />
The fields are defined as follows:<br />
<br />
<pre>signup date;hostname;last, first;company; address;city;state;zip;country;referred by;package id;bandwidth overage option;if gobut is selected, $ amount to limit is listed here;payment method;source ip;template id;<br />
email;phone #;admin;billing;alt<br />
new/replace system;create new account/add to existing;hostname of server being replaced;new IP pref;update/merge contacts;update/merge owner info;update/preserve traffic overage;</pre><br />
<br />
When a customer signs up an email will be sent to support@johncompanies.com containing the log entry in the format above<br />
<br />
The process for setting up a new customer is as follows:<br />
<br />
1. Go to the pending customers screen (choose Mgmt site -> New Signups). New customers will be at the bottom of the "VPS Signups" and "Colo Signups" sections. If they’ve paid automatically (via PayPal) the word “PAID” will appear in the “pmt status” column. The other thing you’ll see in this column is “paid- not cleared”- this means they paid with eCheck (via PayPal) and the check has not cleared. This normally takes 3 biz. days and we wait till it clears and shows “PAID” before setting up the server. Customers who pay with a credit card will not appear as "PAID" here. In that case, an email will be sent to support@johncompanies.com indicating a payment has been made (or failed). Once the payment has been paid/cleared, click “process….” <br />
<br />
Before setting someone up, you need to decide whether the signup is fraudulent. Examples of fraud can be found in /usr/local/www/jc_pub/data/fraud-examples <br />
Generally, <br />
* NOT FRAUD = non-anonymous email accounts that match who they are (dan@brockman.com, where the person is Dan Brockman)<br />
* NOT FRAUD = referred by a customer that actually exists or by kuro5hin<br />
* MAYBE FRAUD = referred by ‘web search’ or ‘google’<br />
* MAYBE FRAUD = generic sounding address<br />
* MAYBE FRAUD = IP address is out of country and address is in US (use dnsstuff link on main signups screen to lookup where IP is)<br />
* FRAUD = they don’t type in: referred by and/or hostname and/or give the default answer for bandwidth is stop<br />
* FRAUD = multiple orders each from the same IP and/or using same email address<br />
* FRAUD = no hostname or hacker-sounding hostname: 3v1l0n3 (evilone)<br />
<br />
If you’re ever in doubt, call the card owner using the phone number given at signup or call the bank number provided for the card. Never use the email as it’s likely the thief’s and not the cardholder’s.<br />
<br />
Almost all the info needed is pre-entered into this screen, but some fields will need attention:<br />
<br />
* System: The system is already selected based on the template/OS the customer selected. <br />
* Directory: information provided for you to enter once the system is created. (leave bank for managedcolo)<br />
* Disk: indicates how much disk space the server should have. (leave bank for managedcolo)<br />
* Hostname: self-explanatory. <br />
* veid: (linux systems only) should be filled in with the significant digits of the customer ID, ex: col01340 = veid 1340. <br />
* os: indicates which OS the customer wants (feed to linux vm script) <br />
* ip(s): for linux customers, you should click “ipmap” and scroll down to the first available (green) ip amongst others on the same system. Click on the ip to copy it back to the “ip(s)” field on the form. For FreeBSD customers, the ips available for use are already assigned to the system and should be copied back to this screen once the jail is made (see below). For colo customers, choose an IP appropriate to the data center. i.e. for i2b pick an IP from the 229 block.<br />
* start date: should reflect the day the system was created. <br />
* asset tag: colo only<br />
* password: VPS only<br />
* cabinet: colo only<br />
* ats port: colo @ i2b only<br />
* monitored: should only be checked if the system’s ip/services were supplied to castle to place on the monitor (probes) list. <br />
<br />
NOTE: you must use a JavaScript enabled browser to enter new customers otherwise you won’t be given correct options for os templates.<br />
<br />
3. create the new system using either [[VPS_Management#jailmake|jailmake]] or [[VPS_Management#vm|vm]] scripts. For posterity, on our older linux systems and older OS's (pre virt17) we used to use a custom script for each OS version: vemakecentos3 vemakedebian40 vemakedebian30 vemakedebian31 vemakefedora2 vemakefedora6 vemakefedora7 vemakesuse100 vemakesuse93 vemakecentos4 vemakedebian31 vemakefedora4 vemakerh9 vemakeubuntu5 vemakefedora vemakefedora5 vemakesuse vemakeubuntu606 vemakeubuntu610 vemakeubuntu704<br />
<br />
[[VPS_Management#jailmake|jailmake]] and [[VPS_Management#vm|vm]] both email the new customer their welcome email.<br />
<br />
When you are done adding a customer, for both systems: copy back the dir and password (supplied by the VPS make script) into the form. For FreeBSD, copy back the IP which the make script will give back to you.<br />
<br />
When you are done filling out all the fields in the pending customer form, click “Activate”. This will create the customer in our database and remove them from the pending list. Nothing is emailed to the customer as a result of this action. If the customer paid via credit card, their info needs to be added manually.<br />
<br />
Discussion about choosing IPs ...<br />
<br />
When you make a new system, you have to choose what IP to give it. New FreeBSD servers are configured with a set amount of IPs, and as you add new systems to that freebsd server you can use one of the IPs assigned to the host, but not currently assigned to a VPS. You can use the js program to see what IPs are available for assigning to new customers. <br />
<br />
However, it is not that simple with the linux systems. The linux servers do not get the IPs of their customer systems bound to the actual machine. That is, even on a fully loaded linux system, if you run `ifconfig -a` from the base machine, you only see one IP - the main IP of that system. Further complicating matters is that linux systems can bind multiple IPs - therefore it is not possible to know that the next new linux system should just have the next IP as the last new one that was created.<br />
<br />
So, what we do is, new linux systems are simply assigned a starting IP, and no new machine (freebsd or linux) is assigned a base IP anywhere within 92-128 Ips of that IP. So you have 92-128 IPs to grow with for the linux systems that will live on that machine.<br />
<br />
So, you simply start creating systems from that starting point – and additional IPs that those systems need should also be added from that range, and even other linux systems on other virt machines should get new IPs from that block of IPs. To find the block of IPs in use, go thru bash history to find last ip used or assigned to a ve. Then, click the “ipmap” link (or pull up Mgmt. -> Reference -> IP Map). Ip’s not already assigned to a machine will show up as green. In general, we want to use ips which have never been assigned, or were relenquishsed years ago (or at least 30 days ago – IMPORTANT!!). <br />
D<br />
If a customer has selected an IP package – is paying for extra IPs now, then we’re pretty much obliged to assign all those IPs now. Otherwise, we only ever assign 1 IP.<br />
<br />
= The Welcome Emails =<br />
<br />
[[VPS_Management#jailmake|jailmake]] and [[VPS_Management#jailmake|vm]], as one of its arguments, asks for an email address. This email address is used to send a welcome email to the new user. However, the welcome email is not inside the jailmake/vemake script, and they are not on the jails themselves either. Here are all the various welcome emails we have:<br />
<br />
<pre>/usr/local/www/jc_pub/data/welcome-freebsdp (dynamic freebsd welcome email, 7.x 8.x)<br />
/usr/local/www/jc_pub/data/welcome-linux (dynamic linux welcome email)<br />
/usr/local/www/jc_pub/data/welcome-debian (dynamic debian/ubuntu welcome email)<br />
/usr/local/www/jc_pub/data/welcome-fedora (dynamic fedora/centos welcome email)<br />
/usr/local/www/jc_pub/data/welcome-freebsd (original, generic freebsd welcome email)<br />
/usr/local/www/jc_pub/data/welcome-freebsd6 (for freebsd6)<br />
/usr/local/www/jc_pub/data/welcome-freebsd7z (for freebsd7 with zfs DEPRECATED)</pre><br />
<br />
on the main johncompanies server.<br />
<br />
When the jailmake script is run, it issues the `fetch` (or wget) command to retrieve it, i.e.<br />
http://www.johncompanies.com/colocation/data/welcome-freebsdp<br />
<br />
and saves it as a temp file, mails it off.<br />
<br />
Welcome emails not noted as dynamic above, are emailed with the IP appended at the top to the email address specified. The password to these accounts was/is generic- not very good. Welcome emails noted as dynamic are parsed by the make script to include the IP and a generated, random root password. <br />
<br />
IMPORTANT NOTE: when creating an older-OS ve on a virt- this almost never happens (i.e. it’s created using the old-style vemakexxxx command), you must use the support address as the email address as the make script will be pulling down the dynamic welcome email when it expects to see the old, generic format and it won’t look good. When you receive the welcome email in support, format properly and resend to customer.<br />
<br />
This means three things:<br />
<br />
a) In order for jailmake to work, the johncompanies web server needs to be up and running, and that file needs to exist at that URL<br />
<br />
b) If you want to edit the welcome email, you only need to edit it on the web server, in one place<br />
<br />
c) If a customer for some reason does not receive the welcome email, then you need to go Mgmt -> Reference -> (file), copy and paste it into an email to them. For the old-generic emails, make sure to add this line to the very top:<br />
<br />
IP: (their IP)<br />
<br />
If the welcome email does not exist, jailmake will still work, in the sense that it will create the system, but no welcome email will be sent.<br />
<br />
= Dedicated Server Setup =<br />
<br />
Before starting the OS install you'll need to know the following pieces of info:<br />
<br />
*CID:<br />
*Customer has multiple colos: Y/N<br />
*Asset tag: JC-xxxx<br />
*rack/location: <br />
*Service/Package (and any deviations to B/W, nfs space, IPs, etc) and price:<br />
*RAM in system:<br />
*IPs included in plan:<br />
*OS (32 vs 64bit variant):<br />
*Hostname:<br />
*Disk partitions, including swap space:<br />
*Number of initial IPs to assign:<br />
*Timezone:<br />
<br />
Some to most of that info will be provided via the new signup page, assuming the customer ordered the server via our order page. The rest will comes from the sales/build department.<br />
<br />
<br />
Once the server is built, it should be installed in the rack, booted to the BIOS screen and labeled with the asset tag and the customer's CID (if available/established).<br />
<br />
If an [[IPKVM]] is not already attached, you will need to ascertain which one is available (look in ~user/kvm, usually open for editing in the p4 screen of the mailbox window).<br />
<br />
Before loading the server, there's a couple things to do in the BIOS screen:<br />
# make sure the date and time are set to UTC<br />
# make sure the server's power restore action is set to: last state (basically what we're trying to do here is if the server has power pulled, when power is restored it should turn back on- assuming it was on when power was pulled. This is what allows our ATS power cycling to work. If this is not set, when the ATS port is turned off and turned back on, the server may not turn back on.<br />
# set the boot order to disable network booting or other things that may slow down bootup. <br />
<br />
After saving and exiting the BIOS, if the server has a raid card installed, you will see the raid BIOS screen. You should enter this screen and setup a raid mirror, or whatever the customer has requested (if a special request was made sales will let you know). Usually we use a 3ware card, to enter the config screen, press ALT-3. Use spacebar to select the 2 drives, tab to 'create unit'. Use defaults. Create a raid1 mirror (unless customer asks for something else). Do not enable write cache, assuming no battery exists. F8 to save and exit.<br />
<br />
Reboot and load the OS- follow the instructions for pulling the ISO into the IPKVM and booting to it [[IPKVM|here]].<br />
<br />
The install should be for the server version of whatever OS was requested. We typically do not install anything other than an sshd (and ports (tree) for FreeBSD). We don't install a GUI environment. We do not setup auto updates. We do not encrypt home directories. <br />
<br />
When selecting and IP, take into consideration the data center: an install at i2b should use an IP from the IPs routed to i2b, and vice versa. A customer's package may come with multiple IPs, however unless they indicate they want more assigned initially (and show good reason for doing so) we only assign 1 IP. If they are to receive more IPs, you will/may need to configure those post-install (depending on the OS). Ideally all IPs they receive are on the same class C block.<br />
<br />
When picking a NIC (most of the installs we do are network versions and require the network to download OS components), it's somewhat of a crap shoot as to which NIC to choose and configure. Usually we try to begin pinging the IP we assign and watch for it to respond immediately after configuring the NIC. If it does not ping, you can:<br />
# ask the NOC to swap the network cable to the alternate port<br />
# go back and configure the alternate NIC (you may have to restart the OS install if it doesn't properly unconfigure the initial NIC, or you can't change it's config to alleviate an IP conflict)<br />
<br />
We set DNS: 69.55.229.3, 69.55.225.225 (for a server @ i2b), 69.55.225.225, 69.55.230.3 (for a server @ castle)<br />
<br />
We setup a regular user account: 'user' and we set that password to 'newrootNNNN' where NNNN is the sig digits of the customer's CID. i.e. for col01233 the password is 'newroot1233'. In FreeBSD, make sure to add 'user' to the wheel group (member of). <br />
If a root password is allowed to be set, we use the same password: 'newrootNNNN' <br />
<br />
Once the OS is installed and you've rebooted (after disconnecting the ISO from the KVM) you should confirm all is working: swap, ram, disk partitions, network is as it should be. If you were only able to assign 1 DNS server in setup, please add another to /etc/resolv.conf:<br />
<br />
@i2b: 69.55.229.3, 69.55.225.225<br />
@castle: 69.55.225.225, 69.55.230.3<br />
<br />
Assuming they have a 3ware raid card, you will need to transfer in the raid CLI tool and our handy script so the customer may check on the health of their raid array, as instructed/encouraged by the welcome email they will receive.<br />
<br />
This package is located on mail;<br />
<pre><br />
fetch http://johncompanies.com/tools/3ware/L64.tgz<br />
tar xvzf L64.tgz<br />
</pre><br />
<br />
The version you choose will of course depend on the OS/version you're installing. You should copy the tarball to the /usr/local/sbin dir (or something in their path) and untar'd. You should run the script included in the tarball called checkraid.sh. If it gives an error, this is due to the fact that on some servers the raid card is recognized as 'c1' and on others 'c0' or something different. Determine which it is for this server by running:<br />
tw_cli info<br />
<br />
which will give you output like:<br />
<pre>Ctl Model (V)Ports Drives Units NotOpt RRate VRate BBU<br />
------------------------------------------------------------------------<br />
c1 9650SE-8LPML 8 6 1 0 5 1 -<br />
</pre><br />
<br />
In this example it's 'c1' so edit checkraid.sh and change all instances of 'c0' to 'c1'<br />
<br />
If we're adding additional IPs (and were not able to do so during initial install), configure as follows:<br />
<br />
Centos/Fedora:<br />
<br />
<pre><br />
cd /etc/sysconfig/network-scripts/<br />
cp -p ifcfg-eth* ~<br />
fetch http://johncompanies.com/tools/network/centos/ifcfg-eth.tgz<br />
tar xvzf ifcfg-eth.tgz<br />
<br />
vi ifcfg-eth0:<br />
<br />
DEVICE=eth0<br />
TYPE=Ethernet<br />
ONBOOT=yes<br />
NM_CONTROLLED=yes<br />
BOOTPROTO=none<br />
IPADDR=69.55.231.XXX # change this to main IP address<br />
PREFIX=24<br />
GATEWAY=69.55.231.1 # change this to 229.1 if IP addr in other block<br />
DNS1=69.55.229.3<br />
DNS2=69.55.225.225<br />
DNS3=69.55.230.3<br />
DOMAIN="johncompanies.com"<br />
DEFROUTE=yes<br />
IPV4_FAILURE_FATAL=yes<br />
IPV6INIT=no<br />
NAME="System eth0"<br />
</pre><br />
<br />
<pre><br />
vi eth0:*<br />
<br />
DEVICE=eth0:0<br />
TYPE=Ethernet<br />
ONBOOT=yes<br />
NM_CONTROLLED=yes<br />
BOOTPROTO=none<br />
IPADDR=69.55.231.XXX # change this to IP address<br />
PREFIX=24<br />
IPV4_FAILURE_FATAL=yes<br />
IPV6INIT=no<br />
</pre><br />
<br />
Repeat above as needed for each IP address<br />
<br />
Remove the un-needed files.<br />
<pre><br />
rm ifcfg-eth0:[3-8] # change numbers to fit how many IP addresses are needed<br />
</pre><br />
<br />
Debian/Ubuntu:<br />
<br />
<pre><br />
cd /etc/network<br />
vi interfaces<br />
<br />
Old file was:<br />
# The loopback network interface<br />
auto lo<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
auto eth0<br />
iface eth0 inet static<br />
address 69.55.227.4<br />
netmask 255.255.255.0<br />
network 69.55.227.0<br />
broadcast 69.55.227.255<br />
gateway 69.55.227.1<br />
# dns-* options are implemented by the resolvconf package, if installed<br />
dns-nameservers 69.55.225.225<br />
dns-search yeppernet.com<br />
<br />
new file is:<br />
# The loopback network interface<br />
auto lo<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
auto eth0<br />
iface eth0 inet static<br />
address 69.55.227.44<br />
netmask 255.255.255.0<br />
network 69.55.227.0<br />
broadcast 69.55.227.255<br />
gateway 69.55.227.1<br />
# dns-* options are implemented by the resolvconf package, if installed<br />
dns-nameservers 69.55D.225.225<br />
dns-search yeppernet.com<br />
<br />
auto eth0:0<br />
iface eth0:0 inet static<br />
address 69.55.227.55<br />
netmask 255.255.255.0<br />
network 69.55.227.0<br />
broadcast 69.55.227.255<br />
gateway 69.55.227.1<br />
<br />
</pre><br />
/etc/init.d/networking restart<br />
(make sure you have console in case you screw that up)<br />
<br />
<pre>ifconfig<br />
eth0 Link encap:Ethernet HWaddr 00:30:48:28:9d:50<br />
inet addr:69.55.227.44 Bcast:69.55.227.255 Mask:255.255.255.0<br />
inet6 addr: fe80::230:48ff:fe28:9d50/64 Scope:Link<br />
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br />
RX packets:2850149 errors:0 dropped:0 overruns:0 frame:0<br />
TX packets:2595566 errors:0 dropped:0 overruns:0 carrier:0<br />
collisions:0 txqueuelen:100<br />
RX bytes:1193068127 (1.1 GB) TX bytes:2046984105 (1.9 GB)<br />
Base address:0x3000 Memory:fc400000-fc420000<br />
<br />
eth0:0 Link encap:Ethernet HWaddr 00:30:48:28:9d:50<br />
inet addr:69.55.227.55 Bcast:69.55.227.255 Mask:255.255.255.0<br />
inet6 addr: fe80::230:48ff:fe28:9d50/64 Scope:Link<br />
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br />
RX packets:2850149 errors:0 dropped:0 overruns:0 frame:0<br />
TX packets:2595566 errors:0 dropped:0 overruns:0 carrier:0<br />
collisions:0 txqueuelen:100<br />
RX bytes:1193068127 (1.1 GB) TX bytes:2046984105 (1.9 GB)<br />
Base address:0x3000 Memory:fc400000-fc420000<br />
</pre><br />
<br />
Ubuntu 18.04<br />
<br />
Ubuntu 18.04 uses netplan instead of /etc/network/interfaces.<br />
<br />
<pre><br />
vi /etc/neplan/50-cloud-init.yaml<br />
# This file is generated from information provided by<br />
# the datasource. Changes to it will not persist across an instance.<br />
# To disable cloud-init's network configuration capabilities, write a file<br />
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:<br />
# network: {config: disabled}<br />
network:<br />
version: 2<br />
renderer: networkd<br />
ethernets:<br />
enp0s25:<br />
dhcp4: no<br />
addresses: [69.55.229.26/24]<br />
gateway4: 69.55.229.1 <br />
nameservers:<br />
addresses: [69.55.229.3,8.8.8.8,8.8.4.4]<br />
<br />
</pre><br />
<br />
The very last thing you should do before logging off the server is to run:<br />
history -c<br />
in every shell you were in so they don't see what we were doing :)<br />
<br />
Assuming this server is connected to an ATS, you'll want to confirm the server is connected to the port you think it is, and test the functionality- that you can power cycle the port and the server will come back up when power is restored. You should do a test power cycle (via the mgmt or AM-based ATS control) while the server is in post or BIOS. Basically, just NOT while the OS is running and disks mounted.<br />
<br />
You should confirm the switch ports to which the server is connected (maybe has a 2nd port for the RMM). While the server is rebooting during your ATS test, you can observe output on the switch console that looks like:<br />
<pre>.Mar 7 09:47:45 PST: %LINK-3-UPDOWN: Interface FastEthernet0/14, changed state to down<br />
.Mar 7 09:47:46 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/14, changed state to down<br />
.Mar 7 09:47:47 PST: %LINK-3-UPDOWN: Interface FastEthernet0/14, changed state to up<br />
.Mar 7 09:47:48 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/14, changed state to up<br />
</pre><br />
This confirms the server is on port 14.<br />
<br />
Take this info (ATS, switch ports plus the asset tag) and:<br />
# activate (or add) the server. Update the cabinet, ATS and port, asset tag. (Switch port not kept in mgmt)<br />
# update the [[cabinetmap|cabinet map]] to add the server to the correct location, along with the ATS port, and switch port(s)<br />
# login to the switch serial console and add the customer's CID (and hostname if they have mult. servers) to the port(s) description tag. Repeat for the RMM port if they have one:<br />
<pre>switch-p25#conf t<br />
Enter configuration commands, one per line. End with CNTL/Z.<br />
switch-p25(config)#int fa0/10<br />
switch-p25(config-if)#des col01233 myhostname<br />
switch-p25(config)#int fa0/11<br />
switch-p25(config-if)#des col01233 myhostname (RMM)<br />
switch-p25(config-if)#end<br />
switch-p25#<br />
.Mar 7 11:37:17 PST: %SYS-5-CONFIG_I: Configured from console by console<br />
switch-p25#wr me<br />
Building configuration...<br />
[OK]<br />
switch-p25#<br />
</pre><br />
<br />
At this point it's safe to hand the server over to the customer. Use the 'new colo welcome' paste. Edit for:<br />
# IP(s)<br />
# if it's FreeBSD use the 'root' & 'user' password line, if it's Ubuntu use/edit the 'root' password line, depending on if you set the root password or not, or what you setup for the normal user account<br />
# the correct DNS depending on which data center they're at (remove ns3c if the server is at castle)<br />
# if they have no raid card, remove that section<br />
# if they have an RMM, include the RMM paste somewhere in the email<br />
<br />
If they're already a customer, you can exclude the info about their AM login info (since they have it already). If they're not, you'll need to reset their AM pass to see it.<br />
<br />
Lastly, assuming this server was installed for a new customer and the customer was not yet activated, the CID was probably not established and the server has no CID label. You'll need to make a note to label it (and add the hostname too if they have multiple servers) next time you're at the data center, or ask the noc staff to do it for you.<br />
<br />
== Dedicated Server Install Checklist ==<br />
<br />
o Install OS<br />
<br />
o Add JCI - adduser, wheel, visudo<br />
<pre><br />
adduser -u 955 jci<br />
passwd jci<br />
''l4mph0st''<br />
adduser -u 1000 user<br />
passwd user <br />
''newrootXXXX'' (XXXX is their col0XXXX number)<br />
vi /etc/group<br />
( add "jci,user" to group wheel and/or sudo )<br />
visudo<br />
( enable group wheel to do anything with password )<br />
</pre><br />
<br />
o Install openssh-clients (yum install openssh-clients)<br />
<pre><br />
yum -y install openssh-clients<br />
</pre><br />
<br />
o Update OS<br />
<pre><br />
yum -y update (for CentOS)<br />
</pre><br />
<br />
o checkraid.sh<br />
As noted above install our RAID disk checks<br />
<br />
o config IP addresses<br />
<br />
o clear last logins<br />
<pre><br />
cp /dev/null /var/log/wtmp<br />
</pre><br />
<br />
o clear history<br />
<pre><br />
cp /dev/null ~/.bash_history<br />
history -c<br />
shutdown -h now<br />
</pre><br />
<br />
o For a suspected spammer, add rule to firewall2 to limit output for mail ports (ie.)<br />
before rule 50<br />
<pre><br />
ipfw add 37 pipe 8 ip from 69.55.231.104/30 to any dst-port 25,465,587<br />
ipfw add 37 pipe 8 ip from 69.55.231.108 to any dst-port 25,465,587<br />
</pre><br />
<br />
=== After installed in rack ===<br />
<br />
check ssh<br />
label switch port<br />
update cabinet map<br />
update mgmt cabinet number<br />
<br />
== RMM Setup ==<br />
<br />
The RMM must be configured via the server's BIOS screen:<br />
<br />
Server management > BMC LAN config:<BR><br />
down to Intel RMM3 LAN config: <BR><br />
IP (static) , mask, g/w<BR><br />
<br />
user config: <BR><br />
disable anonymous. <BR><br />
root: set pass to newrootNNNN</div>75.80.128.147https://wiki.jcihosting.com/index.php?title=VPS_Management&diff=2217VPS Management2020-03-24T00:55:42Z<p>75.80.128.147: /* FreeBSD 10.1 VPS */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
== Moving a server to a different account in the management system ==<br />
<br />
This is a tough one. You gotta touch a bunch of tables in the mysql database.<br />
<pre><br />
systems<br />
subs<br />
ipmap<br />
</pre><br />
<br />
basically, get the sysid and go off that:<br />
<br />
<pre><br />
update systems set cid='newcid' where sysid=xxxxx<br />
update subs set cid='newcid' where sysid=xxxxx<br />
update ipmap set cid='newcid' where sysid=xxxxx<br />
</pre><br />
<br />
You then need to rename the server disk space to change the old ID to the new one.<br />
<br />
For a jail -<br />
<pre><br />
stopjail <hostname><br />
mv 69.55.2XX.XXX-col0<oldid> 69.55.2XX.XXX-col0<newid><br />
mv 69.55.2XX.XXX-col0<oldid>-DIR 69.55.2XX.XXX-col0<newid>-DIR<br />
vi /usr/local/jail/bin/quad1 [Update the ID in the quad file]<br />
startjail <hostname><br />
</pre><br />
<br />
For a virt -<br />
You will have to <br />
* stop the virt <br />
* rename the directories <br />
* modify the config file /vzconf/<veid>.conf <br />
* rename the virt conf file <br />
* restart the new virt.<br />
<br />
that ought to do it..<br />
<br />
<br />
== Re-enable a customer account in the management system ==<br />
<br />
For example to re-instate col03438, In mysql:<br />
<br />
<pre><br />
show tables ;<br />
select * from customers where cid='col03438' ;<br />
update customers set status=1 where cid='col03438' ;<br />
select * from systems where cid='col03438' ;<br />
update systems set stop_date=NULL where cid='col03438' ;<br />
update systems set cancel_date=NULL where cid='col03438' ;<br />
update systems set status=1 where cid='col03438' ;<br />
select * from subs where cid='col03438' ;<br />
update subs set stop_date=NULL where cid='col03438' ;<br />
update subs set status=1 where cid='col03438' <br />
<br />
</pre><br />
<br />
= Ganeti VPS =<br />
<br />
This is how to create a VPS on the Lamphost ganeti cluster.<br />
I have created scripts for creating a FreeBSD VPS and one for<br />
creating a Linux VPS.<br />
<br />
<br />
To create a linux VPS use<br />
<pre><br />
/root/new <name> <disk size> <memory> <IP Addr> [<OS>]<br />
<br />
( for example /root/new vps.example.com 30 2 69.55.238.10 centos-8 )<br />
</pre><br />
<br />
To create a FreeBSD VPS, we have to do a full install. This will create a <br />
new VPS.<br />
<br />
<pre><br />
/root/newfreebsd12 <name> <disk size> <memory> <IP Addr> [<OS>]<br />
</pre><br />
<br />
You will then have to do the install via VNC.<br />
<br />
You need to customize the install by adding a "jci" account and a customer<br />
account (col0XXXX) as well. <br />
<br />
<br> <br />
You will need to install sudo.<br />
<pre> <br />
pkg install sudo # install sudo package<br />
visudo # to add wheel account to sudoers<br />
vi /etc/group # add jci and user acct to wheel<br />
</pre><br />
<br />
<br><br />
You will also have to set up a console in /etc/ttys.<br />
<pre><br />
vi /etc/ttys<br />
</pre><br />
<br />
= FreeBSD 10.1 VPS (jail5 and jail6) =<br />
<br />
This is information on how to manage the FreeBSD 10.1 VPS servers on Jail5.<br />
<br />
<br />
== Making new customer VPS ==<br />
<pre><br />
virtmake IP[,IP] <CID> hostname ipfw# email {FSP|FB|FS|FM|FP} [size in GB]<br />
</pre><br />
<br />
Note: <br />
For now, CID must be unique in the last 2 digits of the CID.<br />
<br />
Must do a "vm start <CID>" after creating a VPS.<br />
<br />
== Starting VPS ==<br />
To start all vms on the system<br />
<pre><br />
vm start<br />
</pre><br />
<br />
To start a single VPS <br />
<pre><br />
vm start <CID><br />
</pre><br />
<br />
== List running VPS ==<br />
<pre><br />
vm status<br />
</pre><br />
<br />
== Attach to a user VPS console ==<br />
<pre><br />
vm attach <CID><br />
</pre><br />
<br />
<br />
To detach from a console, type tilde control-d (~ ctl-d).<br />
<br />
== Stop a VPS ==<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
== Restart a VPS ==<br />
<pre><br />
vm restart <CID><br />
</pre><br />
<br />
<br />
== Modify a VPS ==<br />
<br />
To modify a VPS config, stop the VPS<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
Mount the VPS data file<br />
<pre><br />
vm mount <CID><br />
<br />
cd /data/vm/private/<CID>/mount<br />
</pre><br />
<br />
Modify the user files necessary.<br />
<br />
Restart the VPS.<br />
<br />
<pre><br />
vm umount <CID><br />
<br />
vm start <CID><br />
</pre><br />
<br />
== Notes on a new FBSD 10.x VPS (jail6) ==<br />
<br />
<pre><br />
#! /bin/sh<br />
#<br />
# new <colo#> <disk><br />
<br />
<br />
memory=1024<br />
<br />
x="$1"<br />
<br />
disk="$2"<br />
if [ "y" == "y$disk" ] ; then<br />
disk=60<br />
fi<br />
<br />
disk2=`expr $disk - 4`<br />
<br />
<br />
vm image provision 04dcdf96-d843-11e6-b131-001e67fcc64f col0$x<br />
# vm configure col0$x<br />
sed -e "s/memory=256M/memory=${memory}M/" /vms/col01114/col01114.conf >/vms/col0$x/col0$x.conf<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
# vm start col0$x<br />
<br />
<br />
exit<br />
<br />
vm console col0$x<br />
<br />
#<br />
# to increase the customer disk<br />
#<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
<br />
## on the customer VPS<br />
<br />
gpart recover vtbd0<br />
gpart show vtbd0<br />
swapoff /dev/vtbd0p3<br />
gpart delete -i 3 vtbd0<br />
gpart show vtbd0<br />
sysctl kern.geom.debugflags=16<br />
gpart resize -i 2 -a 4k -s $disk2\G vtbd0<br />
<br />
growfs /dev/vtbd0p2<br />
<br />
gpart show vtbd0<br />
gpart add -t freebsd-swap -a 4k vtbd0<br />
gpart show vtbd0<br />
swapon /dev/vtbd0p3<br />
<br />
df -h<br />
reboot<br />
<br />
vi /etc/rc.conf<br />
hostname<br />
IP Addr<br />
passwd root<br />
passwd user<br />
clear history<br />
cp /dev/null /var/log/utx.log<br />
cp /dev/null ~root/.history<br />
history -c<br />
<br />
</pre><br />
<br />
= FreeBSD Jails =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Making new customer jail 4.x ==<br />
<br />
1. run [[#js|js]] to figure out which partition, IP to put it on, find an unused vn, and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vnX hostname shorthost quad/safe# ipfw# email [gb disk, default 4]</tt><br />
<br />
jail14# /tmp/jailmake 69.55.226.152 /mnt/data2/69.55.226.152-col00182 vn23 negev.cerastes.org negev 4 182 cerastes@cerastes.org<br />
<br />
<br />
== Making new customer jail 6.x ==<br />
<br />
1. run [[#js|js]] to figure out which gvinum vols are free (and which mnt (disk) each gvinum goes with/to), IP to put it on and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vN[,vN] hostname shorthost quad/safe# ipfw# email</tt><br />
<br />
jail19# jailmake 69.55.236.17 /mnt/data1/69.55.236.17-col01574 v10 uncle-leo.commadev.com uncle-leo 1 1574 lee@commadev.com,lee@gmail.com<br />
<br />
<br />
== Changing an ip for freebsd VPS ==<br />
<br />
*stopjail (hostname)<br />
*on systems using vnfile or mdfile: rename vnfile/mdfile, using new ip<br />
*rename dir using new ip<br />
*Edit quad (make sure to update all lines: directory, vn/mdfile)<br />
*buildsafe (not necessary on systems having only quad1)<br />
*if you're adding an ip not already on the host machine: <tt>ipadd x.x.x.x</tt><br />
*startjail (hostname)<br />
if backups: <tt>mvbackups</tt> ...<br />
*Edit dir/ip in mgmt<br />
*if any rules, update any firewall rules to use new ip (careful not to make sure there arent multiple rules per ipfw#- search by colo#)<br />
*(if customer asks and has domains) update their domains on ns1c<br />
*optional: update ipfw counters on the host server<br />
<br />
<br />
== Rename a gconcat vol ==<br />
<br />
*stop jail, unmount everything<br />
*gconcat stop vol<br />
*gconcat clear /dev/gvinum/vN<br />
*gconcat clear /dev/gvinum/vN+1<br />
*gconcat label -v newname /dev/gvinum/vN /dev/gvinum/vN+1<br />
*bsdlabel /dev/concat/newname<br />
(make sure a partition is not fstype unused, if so change to: 4.2BSD)<br />
<br />
<br />
== Remaking a VPS (on same jail) ==<br />
<br />
# take him down (stopjail)<br />
# make note of host, vn/gvin/md, ip<br />
# remove from quad/safe<br />
# move vnfile (rename)<br />
# remove dir (jailmake will make a new one)<br />
# remove ipfw counts<br />
# jailmake<br />
# update backup script<br />
# update db with new dir/ip (use the “move” link, mark old as stopped today)<br />
# update firewall if changed ip<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
<br />
== Adding disk to a 4.x jail ==<br />
<br />
NOTE: if you unmount before vnconfig, re-vnconfig then unmount then vnunconfig<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them.<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to dump to a different partition and create the new vn on the current partition. If no space to make make dump, then move old vn’s to backup or worse, dump to backup.<br />
<br />
<br />
2. <tt>g <customerID></tt> to get the info<br />
<br />
<br />
3. make vnfile, newfs and mount. USE CAUTION not to overwrite existing vn. <br><br />
Ex: <tt>vnconfig –T -S 9g -s labels -c /dev/vn22 /mnt/data2/65.214.160.117-col00496<br><br />
disklabel -r -w vn30 auto<br><br />
newfs /dev/vn30c<br><br />
mkdir /mnt/data2/65.214.160.117-col00496-DIR<br><br />
mount /dev/vn30c /mnt/data2/69.55.237.26-col00241-DIR<br><br />
cd /mnt/data2/65.214.160.117-col00496-DIR</tt><br />
<br />
<br />
4. <tt>jailkill <hostname></tt><br />
<br />
<br />
5. <tt>dump -0a -f - /dev/vn1 | restore -r -f -</tt> <br />
<br />
<br />
6. <tt>rm restoresymtable</tt><br />
<br />
<br />
7. unmount and unvnconfig old system:<br><br />
<tt>umount /dev/vnNNc <br><br />
vnconfig –u /dev/vnNN</tt><br />
<br />
<br />
8. move vnfile. Ex: <tt>mv /mnt/data1/69.55.237.26-col00241 /mnt/data1/old-col00241-vnfile-20110312-noarchive</tt><br />
<br />
<br />
9. remove old dir: <tt>rmdir /mnt/data1/69.55.237.26-col00241-DIR</tt><br />
<br />
<br />
10. edit quad/safe to point to new location, run <tt>[[#buildsafe|buildsafe]]</tt><br />
<br />
<br />
11. start jail: <tt>[[#startjail|startjail]] <hostname></tt><br />
<br />
<br />
12. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
13. mv backups if necessary<br />
<br />
WARNING: if you restore to unmounted vn then you’re actually restoring to /mnt/data1<br />
you can tar and do above to recover<br />
<br />
NOTE: you can’t move a customer to a system that has a base sys that is diff from the one they came from (BSD 4.8 to 4.5 = won’t work)<br />
<br />
NOTE: we don’t mount procfs's to anyone’s system by default.<br />
<br />
NOTE: mount_nulls: <br />
also goofy mount_nulls seen in df -k on jail2<br><br />
also needs to be done in safe and quad<br><br />
if he wants another, use safe/quad as example and add it to file. also exec command at prompt to add mount (no reboot, remount or rejail necessary)<br />
<br />
== Moving customer to another jail machine ==<br />
<br />
Systems should only be moved between similarly-versioned jails. If you cannot get a match, you should go to freebsd.org and see what the differences were between the version you're moving from and the new version. Many of the updates will be to drivers and kernels - these don't affect the customer. Perhaps they updated traceroute or a library...you should copy the new/changed files into the VPS/jail on the new host. An alternate method for moving to a different version would be to buildworld to bring their full distribution up to date. Optional steps for this kind of upgrade are included below.<br />
<br />
1. <tt>g <customerID></tt><br />
<br />
<br />
2. <tt>[[#jailkill|jailkill]] <hostname></tt><br />
<br />
<br />
3. create new device on target system<br><br />
4.x:<br><br />
<pre>vnconfig –T –S 4g –s labels –c /dev/vn1 /mnt/data1/69.55.22x.x-col00XXX<br />
disklabel –r –w vn1 auto<br />
newfs /dev/vn1c</pre><br />
<br />
6.x:<br><br />
<pre>bsdlabel -r -w /dev/gvinum/v1<br />
newfs /dev/gvinum/v1a<br />
or <br />
gconcat label v1-v3 /dev/gvinum/v1 /dev/gvinum/v2 /dev/gvinum/v3<br />
bsdlabel -r -w /dev/concat/v1-v3<br />
newfs /dev/concat/v1-v3a</pre><br />
<br />
7.x+:<br><br />
Run jailmakeempty then skip step 4 & 7 below<br />
<br />
<br />
4. make and mount the dir on new system. <br><br />
Ex: <pre>mkdir /mnt/data2/69.55.230.3-col00123-DIR<br />
chmod 755 /mnt/data2/69.55.230.3-col00123-DIR<br />
mount [device] /mnt/data2/69.55.230.3-col00123-DIR</pre><br />
<br />
<br />
5. [[#stopjail|stopjail]] <hostname> 1<br />
<br />
<br />
6. dump fs to new system. ex: <tt>[[#dumpremoterestore|dumpremoterestore]] /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR</tt> (make sure you can ssh as root on remote machine) <br />
<br />
<br />
6a. OPTIONAL BUILDWORLD:<br><br />
cd /usr/src<br />
make world DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
cd etc<br />
make distribution DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
(you may have to rm an openssh file and re-make dist)<br />
<br />
rm -rf /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/periodic/daily/400.status-disks<br />
vi /etc/periodic/security/100.chksetuid<br />
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt><br><br />
with: <tt>MP='/' (use single quotes)</tt><br />
<br />
Ask user if they want ports overwritten with current, if yes:<br />
cp -r /usr/ports /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/usr<br />
<br />
on source: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d .<br />
(copy in all info)<br />
<br />
on target: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d . <br />
(paste all info)<br />
<br />
on source: <br />
cat /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group<br />
(copy in all info)<br />
<br />
on target:<br />
cat > /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group <br />
(paste all info)<br />
<br />
<br />
7. edit quad on source system copy over entries for the jail to the target system <br />
take care that the vn/gvinum/gconcat/md devices on the target system are’t in use and that the /mnt/dataN path doesn’t need changing<br />
<br />
<br />
8. run [[#buildsafe|buildsafe]] (if this is <=6.x) on target system (copies changes made to quad into safe file)<br />
<br />
<br />
9. remove ip from source system. Ex: <tt>ipdel 69.55.230.3</tt><br />
<br />
<br />
10. add ip to target system. Ex: <tt>ipadd 69.55.230.3</tt><br />
<br />
<br />
11. start new system, with [[#startjail|startjail]] or manually, pasting entries found from running <tt>g <customerID></tt> on the new system<br />
<br />
<br />
12. run <tt>[[#canceljail|canceljail]] col0xxxx</tt> on source system. This should prompt you to remove backups if any existed. Do not let it do this, rather:<br />
<br />
<br />
13. if backups existed move them to the new host via the <tt>[[#mvbackups_.28freebsd.29|mvbackups]] script</tt><br />
<br />
<br />
14. edit quad on source system, edit comment to reflect a move rather than a cancel, ex: <tt># moved to jail2 col00241</tt><br />
<br />
<br />
15. edit mgmt to reflect new host and dir for new system<br />
<br />
<br />
16. optional add ipfw rules on new system. Ex:<br />
<tt>ipfw add 01231 count ip from 69.55.230.2 to any<br />
ipfw add 01232 count ip from any to 69.55.230.2</tt><br />
<br />
== Increasing inodes for a VPS ==<br />
<br />
when doing the newfs: <br />
newfs -i 4096 ...<br />
<br />
<br />
== Jail services unable to contact services within same jail ==<br />
<br />
This is due to a messed up routing table (only seen it on jail2 before). <br />
<br />
route delete 69.55.228.65/32 route add -net 69.55.228.65/32 -iface fxp0 -nostatic -cloning route add 69.55.228.65 -iface lo0<br />
<br />
<br />
== /dev/null permission resets == <br />
<br />
Applies to 4.x jails only<br />
<br />
For some reason that I do not understand at all, the /dev/null node in customer jails, after the system is restarted, often reverts permissions to 0600 ... which is bad, because non-root processes cannot redirect to /dev/null anymore. In fact, a fair number of server daemons will complain and/or fail if this is the case.<br />
<br />
So, after a restart, after all systems have restarted, run postboot, which does something like:<br />
<br />
for f in `df -k | grep /dev/vn | awk '{print $6}'` ; do chmod 0666 $f/dev/null ; done<br />
<br />
Later, after the system is up, if anyone ever complains about their /dev/null permissions for any reason ... really if anyone mentions<br />
/dev/null in any way, tell them to run:<br />
<br />
chmod 0666 /dev/null<br />
<br />
We have ultimately fixed this by adding chmods to the quad/safe's<br />
<br />
<br />
== mknod in a jail == <br />
<br />
For a customer who wants to run a chroot name server (named) in a jail.<br />
<br />
<pre><br />
cd /mnt/data1/<user directory>/var/named<br />
mknod -c 1 1 null<br />
mknod -c 2 3 random<br />
</pre><br />
<br />
== Postfix Problems ==<br />
<br />
This hasn't happened in a long time, probably no longer valid with modern OS's<br />
<br />
Postfix is an alternate MTA - people replace sendmail with it in much the same way that many people also replace sendmail with qmail. The problem is that if you install postfix inside of a jail, by default it will not work properly - the user will see error messages like this:<br />
<br />
incoming mail generates errors like this:<br />
<br />
mail_queue_enter: create file incoming/298266.55902: File too large<br />
<br />
and further, sending mail generates errors like this:<br />
<br />
postdrop: warning: mail_queue_enter: create file maildrop/627930.56676: File too large<br />
<br />
This is very easy to solve. In fact, the FreeBSD welcome email that is sent out by `jailmake` now contains this block of text:<br />
<br />
- if you plan on installing postfix, email us first and ask for a necessary patch.<br />
<br />
So, anytime you see a support email that says anything about postfix not working, paste that line from the welcome email into your response to them, and right away, forward the postfix patch to them in an email.<br />
<br />
Here are the instructions:<br />
<br />
Simply copy the patch into /usr/ports/mail/postfix/files, rename the file to ‘patch-file_limit.c’ and then cd to /usr/ports/mail/postfix and run `make install`.<br />
<br />
The postfix patch is the very first email in the support email box, and we never delete or save it because we always want it there to forward to people. The instructions in the email are very clear, and it solves their problem 100% of the time.<br />
<br />
<br />
== Problems with `find` ==<br />
<br />
There was a bug in the FreeBSD vn-filesystem code. The bug causes the system to crash or hang when someone runs the `find` command inside their vn-backed filesystem (and even though the bug may have been fixed, due to the i/o overhead, we still don't like find).<br />
<br />
Now, this does not always happen - however, any jail machine with more than 20 systems on it will crash every night if every system on it runs the daily periodic script out of their crontab, and updates their locate database with the find command.<br />
<br />
(the locate database is a small, FreeBSD specific database that is populated nightly from a cron job, and is then used to provide fast answers to the `locate` command)<br />
<br />
In normal systems (32-40 jails on them), all that has to be done is make sure nobody runs the daily periodic. This means that the file /etc/crontab, instead of looking like this:<br />
<br />
<pre># do daily/weekly/monthly maintenance<br />
1 3 * * * root periodic daily<br />
15 4 * * 6 root periodic weekly<br />
30 5 1 * * root periodic monthly</pre><br />
<br />
Needs to look like this:<br />
<br />
<pre># DO NOT UNCOMMENT THESE - contact support@johncompanies.com for details<br />
#1 3 * * * root periodic daily<br />
#15 4 * * 6 root periodic weekly<br />
#30 5 1 * * root periodic monthly<br />
## DO NOT UNCOMMENT THESE ^^^^^^^^</pre><br />
<br />
and the problem will generally not occur at all. New filesystem images are always altered in this way, so all systems are set like this.<br />
<br />
However, if you have enough systems running, even casual uses of find, outside of the periodic scripts, can cause the system to crash. We are seeing this on jail13, which crashes about once per week.<br />
<br />
There is not yet any resolution to this problem.<br />
<br />
So, if you have a freebsd system crash, see if anyone has uncommented their daily periodic line - you can see them for everyone by running:<br />
<br />
/bin/sh<br />
<br />
and then running:<br />
<br />
<pre>for f in `df -k | grep vn | awk '{print $6}'` ; do echo `cat $f/etc/crontab | grep "periodic daily"` $f ; done | more</pre><br />
<br />
(all one line, one command)<br />
<br />
And then edit those /etc/crontabs and comment them out again.<br />
<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
* if you are getting disk full messages for a BSD customer, it's fairly safe to clear out their /usr/ports/distfiles dir<br />
* 4.x: ps and top an only be run by root in these jails. Done on purpose:<br />
As for `ps` and `top` - non-root users can run them, just not successfully because we have locked the permissions on /dev/mem and /dev/kmem to be root-readable only. That is why non-root users cannot successfully run `ps` and `top`.<br />
* user quotas do not work on freebsd jails - you cannot set up quotas at all, and that's that.<br />
* You cannot inject a process into a 4.x jail, only HUP running processes. 6.x and onward you can with jexec<br />
* jails see base machine’s uptime/load when running top/w<br />
* if someone is unable to get in- cant ping, etc- see if they were blocked by castle (DoS), see if their ip is on the system (post reboot it was lost cause wasn’t in the rc.conf). preboot should catch that.<br />
* in FreeBSD you cant su to root unless you belong to wheel group – so if you remove your acct and setup a new one, we have to add it to wheel group (add to /etc/group)<br />
* Dmesg from underlying sys is seen in customers dmesg on jail<br />
* Popper process going crazy @ 40% for 10min = someone who leaves mail on server<br />
* Don’t force umounts on 4.x jails – it crashes the machine, generally ok on newer machines (running md)<br />
* Good book for admin http://search.barnesandnoble.com/booksearch/isbnInquiry.asp?userid=t824VyRAYz&isbn=0596005164&itm=2<br />
* Self-sign ssl cert http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#selfcert<br />
* conversation with Glenn about semaphores and pgsql<br />
<pre>SDBoody: hey, these are valid amounts/figures/increments right:<br />
kern.ipc.semmni=1280<br />
kern.ipc.semmns=1280<br />
<br />
gr8feen: probably... I always forget exactly what those are, so I usually have to look them up<br />
SDBoody: semaphores<br />
SDBoody: i took the current 1024 and added 256 to them<br />
SDBoody: need more for pgsql<br />
gr8feen: I meant the mni and mns parts...<br />
gr8feen: some of those are not ones you just want to add to... hang on a sec and I'll look them up..<br />
gr8feen: what's semmsl set to?<br />
SDBoody: kern.ipc.semmsl: 1024<br />
<br />
SDBoody: kern.ipc.msgseg: 2048 <br />
kern.ipc.msgssz: 8 <br />
kern.ipc.msgtql: 40 <br />
kern.ipc.msgmnb: 2048 <br />
kern.ipc.msgmni: 40 <br />
kern.ipc.msgmax: 16384 <br />
kern.ipc.semaem: 16384 <br />
kern.ipc.semvmx: 65534 <br />
kern.ipc.semusz: 152 <br />
kern.ipc.semume: 10 <br />
kern.ipc.semopm: 100 <br />
kern.ipc.semmsl: 1024 <br />
kern.ipc.semmnu: 512 <br />
kern.ipc.semmns: 1024 <br />
kern.ipc.semmni: 1024 <br />
kern.ipc.semmap: 768 <br />
kern.ipc.shm_allow_removed: 0 <br />
kern.ipc.shm_use_phys: 1 <br />
kern.ipc.shmall: 262144 <br />
kern.ipc.shmseg: 256 <br />
kern.ipc.shmmni: 784 <br />
kern.ipc.shmmin: 1 <br />
kern.ipc.shmmax: 536870912 <br />
kern.ipc.maxsockets: 25600 <br />
<br />
gr8feen: ok...msl is max per id, mni is max ids, mns is max number of semaphores... so you probably want something like mns = mni * msl<br />
gr8feen: which one did you run out of?<br />
SDBoody: not sure how to tell- ipcs shows the sems in use add up to 1024<br />
SDBoody: there are 59 entries<br />
gr8feen: I'm assuming you tried to start postgres and it failed?<br />
SDBoody: yes<br />
gr8feen: it should have logged why, somewhere..<br />
gr8feen: if I recall, it'll tell you which one it ran out of<br />
SDBoody: > DETAIL: Failed system call was semget(1, 17, 03600).<br />
<br />
gr8feen: so it wanted an id with 17 semaphores...I'd start by making mns = 17*mni and leave mni and mnl set to what they are now and see what it does<br />
SDBoody: i think mni is plenty high<br />
SDBoody: ok, more reasonable 17408<br />
gr8feen: yeah...just change that one and see how it goes..<br />
SDBoody: and leave mni alone at 1024?<br />
gr8feen: yeah...mni id the max number of ids...but if it's trying to get something like 17 per id, your going to hit mns before you hit anything else<br />
SDBoody: right, but doesn't hurt to have it that high (assuming)<br />
<br />
gr8feen: not really... I think those get allocated out of ram that you cant page out, but it's still such a small amount that it really doesn't matter<br />
SDBoody: looks like that worked, thx!<br />
gr8feen: cool<br />
<br />
see semaphores: ipcs -a -s</pre><br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
== Making new customer VE (vm) ==<br />
<br />
This applies only to new virts >= 4.x<br />
<br />
grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.<br />
<br />
1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
also, ping the IP address.<br />
<br />
2. put CT on whichever partition has more space<br />
df -h<br />
<br />
3. vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> ) <br />
vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@johncompanies.com centos-6-x86_64 LM<br />
<br />
4. copy veid, dir, ip and password to pending customer screen. activate customer<br />
<br />
== Making new customer VE (vemakexxx) ==<br />
<br />
This applies to older virts with old templates. This should probably not be used at all anymore.<br />
<br />
<br />
1. look thru hist for ip<br />
<br />
2. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
<br />
3. put ve on whichever partition has more space<br />
vemakerh9 <veid> <ip> <hostname> <mount> <email> [gb disk]; <256|384|512> <veid><br />
vemakerh9 866 69.55.226.109 ngentu.com /vz1 ayo@ngantu.com,asd@asd.com 5; 256 866<br />
<br />
4. copy (veid), dir, and ip to pending customer screen (pass set to p455agfa)<br />
<br />
<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrateonline/migrate failures (migrate manually) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
You may also see migration failing due to quota issues.<br />
<br />
You can try to resolve by copying any quota file into the file you need:<br />
<br />
cp /var/vzquota/quota.1 /var/vzquota/quota.xxx<br />
<br />
If it complains about quota running you should then be able to stop it<br />
<br />
vzquota off xxxx<br />
<br />
If all else fails, migrate to a new VEID<br />
i.e. 1234 becomes 12341<br />
<br />
<br />
If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
'''Note:''' don't forget to fix /usr/local/sbin/backup.config for new disk location<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
=== Cant change file /etc/sysconfig/network ===<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
=== What template is user using ===<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
=== Edits needed to run java ===<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
=== Set time zone ===<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
=== Failing shm_open calls ===<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
=== Debian can't get on the network ===<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
=== CentOS can't get on the network ===<br />
<br />
When CentOS system was running, but can't get on the network, they probably updated iproute via yum.<br />
<br />
Copy an older version of the iproute rpm to their VPS.<br />
<br />
<pre>cp -p /root/iproute/iproute-2.6.32-23.el6.<i686>.rpm /vz2/private/<1527>/fs/root</pre><br />
<br />
Enter their ve, install rpm, restart network, and prevent yum from updating iproute.<br />
<br />
<pre><br />
ve <1527><br />
rpm -i --force iproute-2.6.32-23.el6.<i686>.rpm<br />
rpm -q iproute<br />
rpm -e <new iproute rpm><br />
service network restart<br />
echo "exclude=iproute" >>/etc/yum/ip-exclude<br />
echo "include=/etc/yum/ip-exclude" >>/etc/yum.conf<br />
</pre><br />
<br />
<br />
<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
----<br />
<br />
* uptime/load that vz users see isn’t what we see- it's their own load<br />
* Vz overwrites resolve.conf on every boot to what’s in ve config file. Also overwrites hostname<br />
* Quotas in vz: set Gid_limit in vz config file and restart ve. Errors leading up to this will be out of files but no inode or space issues. set to 2000 in conf, and restart vz<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
To create a new Linux VPS<br />
<br />
* get an IP address from mgmt screen<br />
* run df -h to figure out which disk has enough space<br />
* run vm with no arguments to get syntax and exact template name<br />
* use col0xxxx for hostname if they don’t give you a hostname<br />
* copy over dir, veid, ip and password to pending customer screen<br />
<br />
vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> )<br />
<br />
== disable ==<br />
To disable a customers Virtuozzo VPS server.<br />
<br />
* stop the server<br />
vzctl stop <veid><br />
* prevent restart on server reboot and user restart via Control Panel<br />
vzctl set <veid> --disabled=yes --offline_management=no --save<br />
<br />
== cancelve ==<br />
When a customer cancels all service.<br />
<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
<br />
display the contents of the vz config file for the <veid><br />
<br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
<br />
List all virtual environments running on this machine.<br />
<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
<br />
show processes running under this <veid>.<br />
<br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
<br />
Restart a <veid>.<br />
<br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
<br />
enter a <veid> as root.<br />
<br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
<br />
execute a single command in the <veid> environment.<br />
<br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem VEID <256|512|768|1024|1280|1536|2048> [burst GB]<br />
adjusts the memory resources for the VE. If this is on VZ ver >4, you will also be prompted to provide a burst ram amount. Sample usage:<br />
<br />
<pre># setmem 21671 1024 4<br />
/usr/sbin/vzctl set 21671 --ram 1024m --swap 3072m --save<br />
</pre><br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>75.80.128.147https://wiki.jcihosting.com/index.php?title=VPS_Management&diff=2216VPS Management2020-03-24T00:53:29Z<p>75.80.128.147: /* Notes on a new FBSD 10.x VPS */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
== Moving a server to a different account in the management system ==<br />
<br />
This is a tough one. You gotta touch a bunch of tables in the mysql database.<br />
<pre><br />
systems<br />
subs<br />
ipmap<br />
</pre><br />
<br />
basically, get the sysid and go off that:<br />
<br />
<pre><br />
update systems set cid='newcid' where sysid=xxxxx<br />
update subs set cid='newcid' where sysid=xxxxx<br />
update ipmap set cid='newcid' where sysid=xxxxx<br />
</pre><br />
<br />
You then need to rename the server disk space to change the old ID to the new one.<br />
<br />
For a jail -<br />
<pre><br />
stopjail <hostname><br />
mv 69.55.2XX.XXX-col0<oldid> 69.55.2XX.XXX-col0<newid><br />
mv 69.55.2XX.XXX-col0<oldid>-DIR 69.55.2XX.XXX-col0<newid>-DIR<br />
vi /usr/local/jail/bin/quad1 [Update the ID in the quad file]<br />
startjail <hostname><br />
</pre><br />
<br />
For a virt -<br />
You will have to <br />
* stop the virt <br />
* rename the directories <br />
* modify the config file /vzconf/<veid>.conf <br />
* rename the virt conf file <br />
* restart the new virt.<br />
<br />
that ought to do it..<br />
<br />
<br />
== Re-enable a customer account in the management system ==<br />
<br />
For example to re-instate col03438, In mysql:<br />
<br />
<pre><br />
show tables ;<br />
select * from customers where cid='col03438' ;<br />
update customers set status=1 where cid='col03438' ;<br />
select * from systems where cid='col03438' ;<br />
update systems set stop_date=NULL where cid='col03438' ;<br />
update systems set cancel_date=NULL where cid='col03438' ;<br />
update systems set status=1 where cid='col03438' ;<br />
select * from subs where cid='col03438' ;<br />
update subs set stop_date=NULL where cid='col03438' ;<br />
update subs set status=1 where cid='col03438' <br />
<br />
</pre><br />
<br />
= Ganeti VPS =<br />
<br />
This is how to create a VPS on the Lamphost ganeti cluster.<br />
I have created scripts for creating a FreeBSD VPS and one for<br />
creating a Linux VPS.<br />
<br />
<br />
To create a linux VPS use<br />
<pre><br />
/root/new <name> <disk size> <memory> <IP Addr> [<OS>]<br />
<br />
( for example /root/new vps.example.com 30 2 69.55.238.10 centos-8 )<br />
</pre><br />
<br />
To create a FreeBSD VPS, we have to do a full install. This will create a <br />
new VPS.<br />
<br />
<pre><br />
/root/newfreebsd12 <name> <disk size> <memory> <IP Addr> [<OS>]<br />
</pre><br />
<br />
You will then have to do the install via VNC.<br />
<br />
You need to customize the install by adding a "jci" account and a customer<br />
account (col0XXXX) as well. <br />
<br />
<br> <br />
You will need to install sudo.<br />
<pre> <br />
pkg install sudo # install sudo package<br />
visudo # to add wheel account to sudoers<br />
vi /etc/group # add jci and user acct to wheel<br />
</pre><br />
<br />
<br><br />
You will also have to set up a console in /etc/ttys.<br />
<pre><br />
vi /etc/ttys<br />
</pre><br />
<br />
= FreeBSD 10.1 VPS =<br />
<br />
This is information on how to manage the FreeBSD 10.1 VPS servers on Jail5.<br />
<br />
<br />
== Making new customer VPS ==<br />
<pre><br />
virtmake IP[,IP] <CID> hostname ipfw# email {FSP|FB|FS|FM|FP} [size in GB]<br />
</pre><br />
<br />
Note: <br />
For now, CID must be unique in the last 2 digits of the CID.<br />
<br />
Must do a "vm start <CID>" after creating a VPS.<br />
<br />
== Starting VPS ==<br />
To start all vms on the system<br />
<pre><br />
vm start<br />
</pre><br />
<br />
To start a single VPS <br />
<pre><br />
vm start <CID><br />
</pre><br />
<br />
== List running VPS ==<br />
<pre><br />
vm status<br />
</pre><br />
<br />
== Attach to a user VPS console ==<br />
<pre><br />
vm attach <CID><br />
</pre><br />
<br />
<br />
To detach from a console, type tilde control-d (~ ctl-d).<br />
<br />
== Stop a VPS ==<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
== Restart a VPS ==<br />
<pre><br />
vm restart <CID><br />
</pre><br />
<br />
<br />
== Modify a VPS ==<br />
<br />
To modify a VPS config, stop the VPS<br />
<pre><br />
vm stop <CID><br />
</pre><br />
<br />
Mount the VPS data file<br />
<pre><br />
vm mount <CID><br />
<br />
cd /data/vm/private/<CID>/mount<br />
</pre><br />
<br />
Modify the user files necessary.<br />
<br />
Restart the VPS.<br />
<br />
<pre><br />
vm umount <CID><br />
<br />
vm start <CID><br />
</pre><br />
<br />
== Notes on a new FBSD 10.x VPS (jail6) ==<br />
<br />
<pre><br />
#! /bin/sh<br />
#<br />
# new <colo#> <disk><br />
<br />
<br />
memory=1024<br />
<br />
x="$1"<br />
<br />
disk="$2"<br />
if [ "y" == "y$disk" ] ; then<br />
disk=60<br />
fi<br />
<br />
disk2=`expr $disk - 4`<br />
<br />
<br />
vm image provision 04dcdf96-d843-11e6-b131-001e67fcc64f col0$x<br />
# vm configure col0$x<br />
sed -e "s/memory=256M/memory=${memory}M/" /vms/col01114/col01114.conf >/vms/col0$x/col0$x.conf<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
# vm start col0$x<br />
<br />
<br />
exit<br />
<br />
vm console col0$x<br />
<br />
#<br />
# to increase the customer disk<br />
#<br />
zfs set quota=${disk}G zroot/vms/col0$x<br />
zfs set volsize=${disk}G zroot/vms/col0$x/disk0<br />
<br />
## on the customer VPS<br />
<br />
gpart recover vtbd0<br />
gpart show vtbd0<br />
swapoff /dev/vtbd0p3<br />
gpart delete -i 3 vtbd0<br />
gpart show vtbd0<br />
sysctl kern.geom.debugflags=16<br />
gpart resize -i 2 -a 4k -s $disk2\G vtbd0<br />
<br />
growfs /dev/vtbd0p2<br />
<br />
gpart show vtbd0<br />
gpart add -t freebsd-swap -a 4k vtbd0<br />
gpart show vtbd0<br />
swapon /dev/vtbd0p3<br />
<br />
df -h<br />
reboot<br />
<br />
vi /etc/rc.conf<br />
hostname<br />
IP Addr<br />
passwd root<br />
passwd user<br />
clear history<br />
cp /dev/null /var/log/utx.log<br />
cp /dev/null ~root/.history<br />
history -c<br />
<br />
</pre><br />
<br />
= FreeBSD Jails =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Making new customer jail 4.x ==<br />
<br />
1. run [[#js|js]] to figure out which partition, IP to put it on, find an unused vn, and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vnX hostname shorthost quad/safe# ipfw# email [gb disk, default 4]</tt><br />
<br />
jail14# /tmp/jailmake 69.55.226.152 /mnt/data2/69.55.226.152-col00182 vn23 negev.cerastes.org negev 4 182 cerastes@cerastes.org<br />
<br />
<br />
== Making new customer jail 6.x ==<br />
<br />
1. run [[#js|js]] to figure out which gvinum vols are free (and which mnt (disk) each gvinum goes with/to), IP to put it on and choose which quad it should go in<br />
<br />
<br />
2. use col00xxx for both hostnames if they don’t give you a hostname<br />
<br />
<br />
3. copy over dir and ip to pending customer screen<br />
<br />
<br />
Usage: <tt>jailmake IP filepath vN[,vN] hostname shorthost quad/safe# ipfw# email</tt><br />
<br />
jail19# jailmake 69.55.236.17 /mnt/data1/69.55.236.17-col01574 v10 uncle-leo.commadev.com uncle-leo 1 1574 lee@commadev.com,lee@gmail.com<br />
<br />
<br />
== Changing an ip for freebsd VPS ==<br />
<br />
*stopjail (hostname)<br />
*on systems using vnfile or mdfile: rename vnfile/mdfile, using new ip<br />
*rename dir using new ip<br />
*Edit quad (make sure to update all lines: directory, vn/mdfile)<br />
*buildsafe (not necessary on systems having only quad1)<br />
*if you're adding an ip not already on the host machine: <tt>ipadd x.x.x.x</tt><br />
*startjail (hostname)<br />
if backups: <tt>mvbackups</tt> ...<br />
*Edit dir/ip in mgmt<br />
*if any rules, update any firewall rules to use new ip (careful not to make sure there arent multiple rules per ipfw#- search by colo#)<br />
*(if customer asks and has domains) update their domains on ns1c<br />
*optional: update ipfw counters on the host server<br />
<br />
<br />
== Rename a gconcat vol ==<br />
<br />
*stop jail, unmount everything<br />
*gconcat stop vol<br />
*gconcat clear /dev/gvinum/vN<br />
*gconcat clear /dev/gvinum/vN+1<br />
*gconcat label -v newname /dev/gvinum/vN /dev/gvinum/vN+1<br />
*bsdlabel /dev/concat/newname<br />
(make sure a partition is not fstype unused, if so change to: 4.2BSD)<br />
<br />
<br />
== Remaking a VPS (on same jail) ==<br />
<br />
# take him down (stopjail)<br />
# make note of host, vn/gvin/md, ip<br />
# remove from quad/safe<br />
# move vnfile (rename)<br />
# remove dir (jailmake will make a new one)<br />
# remove ipfw counts<br />
# jailmake<br />
# update backup script<br />
# update db with new dir/ip (use the “move” link, mark old as stopped today)<br />
# update firewall if changed ip<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
<br />
== Adding disk to a 4.x jail ==<br />
<br />
NOTE: if you unmount before vnconfig, re-vnconfig then unmount then vnunconfig<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them.<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to dump to a different partition and create the new vn on the current partition. If no space to make make dump, then move old vn’s to backup or worse, dump to backup.<br />
<br />
<br />
2. <tt>g <customerID></tt> to get the info<br />
<br />
<br />
3. make vnfile, newfs and mount. USE CAUTION not to overwrite existing vn. <br><br />
Ex: <tt>vnconfig –T -S 9g -s labels -c /dev/vn22 /mnt/data2/65.214.160.117-col00496<br><br />
disklabel -r -w vn30 auto<br><br />
newfs /dev/vn30c<br><br />
mkdir /mnt/data2/65.214.160.117-col00496-DIR<br><br />
mount /dev/vn30c /mnt/data2/69.55.237.26-col00241-DIR<br><br />
cd /mnt/data2/65.214.160.117-col00496-DIR</tt><br />
<br />
<br />
4. <tt>jailkill <hostname></tt><br />
<br />
<br />
5. <tt>dump -0a -f - /dev/vn1 | restore -r -f -</tt> <br />
<br />
<br />
6. <tt>rm restoresymtable</tt><br />
<br />
<br />
7. unmount and unvnconfig old system:<br><br />
<tt>umount /dev/vnNNc <br><br />
vnconfig –u /dev/vnNN</tt><br />
<br />
<br />
8. move vnfile. Ex: <tt>mv /mnt/data1/69.55.237.26-col00241 /mnt/data1/old-col00241-vnfile-20110312-noarchive</tt><br />
<br />
<br />
9. remove old dir: <tt>rmdir /mnt/data1/69.55.237.26-col00241-DIR</tt><br />
<br />
<br />
10. edit quad/safe to point to new location, run <tt>[[#buildsafe|buildsafe]]</tt><br />
<br />
<br />
11. start jail: <tt>[[#startjail|startjail]] <hostname></tt><br />
<br />
<br />
12. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
13. mv backups if necessary<br />
<br />
WARNING: if you restore to unmounted vn then you’re actually restoring to /mnt/data1<br />
you can tar and do above to recover<br />
<br />
NOTE: you can’t move a customer to a system that has a base sys that is diff from the one they came from (BSD 4.8 to 4.5 = won’t work)<br />
<br />
NOTE: we don’t mount procfs's to anyone’s system by default.<br />
<br />
NOTE: mount_nulls: <br />
also goofy mount_nulls seen in df -k on jail2<br><br />
also needs to be done in safe and quad<br><br />
if he wants another, use safe/quad as example and add it to file. also exec command at prompt to add mount (no reboot, remount or rejail necessary)<br />
<br />
== Moving customer to another jail machine ==<br />
<br />
Systems should only be moved between similarly-versioned jails. If you cannot get a match, you should go to freebsd.org and see what the differences were between the version you're moving from and the new version. Many of the updates will be to drivers and kernels - these don't affect the customer. Perhaps they updated traceroute or a library...you should copy the new/changed files into the VPS/jail on the new host. An alternate method for moving to a different version would be to buildworld to bring their full distribution up to date. Optional steps for this kind of upgrade are included below.<br />
<br />
1. <tt>g <customerID></tt><br />
<br />
<br />
2. <tt>[[#jailkill|jailkill]] <hostname></tt><br />
<br />
<br />
3. create new device on target system<br><br />
4.x:<br><br />
<pre>vnconfig –T –S 4g –s labels –c /dev/vn1 /mnt/data1/69.55.22x.x-col00XXX<br />
disklabel –r –w vn1 auto<br />
newfs /dev/vn1c</pre><br />
<br />
6.x:<br><br />
<pre>bsdlabel -r -w /dev/gvinum/v1<br />
newfs /dev/gvinum/v1a<br />
or <br />
gconcat label v1-v3 /dev/gvinum/v1 /dev/gvinum/v2 /dev/gvinum/v3<br />
bsdlabel -r -w /dev/concat/v1-v3<br />
newfs /dev/concat/v1-v3a</pre><br />
<br />
7.x+:<br><br />
Run jailmakeempty then skip step 4 & 7 below<br />
<br />
<br />
4. make and mount the dir on new system. <br><br />
Ex: <pre>mkdir /mnt/data2/69.55.230.3-col00123-DIR<br />
chmod 755 /mnt/data2/69.55.230.3-col00123-DIR<br />
mount [device] /mnt/data2/69.55.230.3-col00123-DIR</pre><br />
<br />
<br />
5. [[#stopjail|stopjail]] <hostname> 1<br />
<br />
<br />
6. dump fs to new system. ex: <tt>[[#dumpremoterestore|dumpremoterestore]] /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR</tt> (make sure you can ssh as root on remote machine) <br />
<br />
<br />
6a. OPTIONAL BUILDWORLD:<br><br />
cd /usr/src<br />
make world DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
cd etc<br />
make distribution DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR<br />
<br />
(you may have to rm an openssh file and re-make dist)<br />
<br />
rm -rf /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/periodic/daily/400.status-disks<br />
vi /etc/periodic/security/100.chksetuid<br />
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt><br><br />
with: <tt>MP='/' (use single quotes)</tt><br />
<br />
Ask user if they want ports overwritten with current, if yes:<br />
cp -r /usr/ports /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/usr<br />
<br />
on source: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d .<br />
(copy in all info)<br />
<br />
on target: <br />
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d . <br />
(paste all info)<br />
<br />
on source: <br />
cat /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group<br />
(copy in all info)<br />
<br />
on target:<br />
cat > /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group <br />
(paste all info)<br />
<br />
<br />
7. edit quad on source system copy over entries for the jail to the target system <br />
take care that the vn/gvinum/gconcat/md devices on the target system are’t in use and that the /mnt/dataN path doesn’t need changing<br />
<br />
<br />
8. run [[#buildsafe|buildsafe]] (if this is <=6.x) on target system (copies changes made to quad into safe file)<br />
<br />
<br />
9. remove ip from source system. Ex: <tt>ipdel 69.55.230.3</tt><br />
<br />
<br />
10. add ip to target system. Ex: <tt>ipadd 69.55.230.3</tt><br />
<br />
<br />
11. start new system, with [[#startjail|startjail]] or manually, pasting entries found from running <tt>g <customerID></tt> on the new system<br />
<br />
<br />
12. run <tt>[[#canceljail|canceljail]] col0xxxx</tt> on source system. This should prompt you to remove backups if any existed. Do not let it do this, rather:<br />
<br />
<br />
13. if backups existed move them to the new host via the <tt>[[#mvbackups_.28freebsd.29|mvbackups]] script</tt><br />
<br />
<br />
14. edit quad on source system, edit comment to reflect a move rather than a cancel, ex: <tt># moved to jail2 col00241</tt><br />
<br />
<br />
15. edit mgmt to reflect new host and dir for new system<br />
<br />
<br />
16. optional add ipfw rules on new system. Ex:<br />
<tt>ipfw add 01231 count ip from 69.55.230.2 to any<br />
ipfw add 01232 count ip from any to 69.55.230.2</tt><br />
<br />
== Increasing inodes for a VPS ==<br />
<br />
when doing the newfs: <br />
newfs -i 4096 ...<br />
<br />
<br />
== Jail services unable to contact services within same jail ==<br />
<br />
This is due to a messed up routing table (only seen it on jail2 before). <br />
<br />
route delete 69.55.228.65/32 route add -net 69.55.228.65/32 -iface fxp0 -nostatic -cloning route add 69.55.228.65 -iface lo0<br />
<br />
<br />
== /dev/null permission resets == <br />
<br />
Applies to 4.x jails only<br />
<br />
For some reason that I do not understand at all, the /dev/null node in customer jails, after the system is restarted, often reverts permissions to 0600 ... which is bad, because non-root processes cannot redirect to /dev/null anymore. In fact, a fair number of server daemons will complain and/or fail if this is the case.<br />
<br />
So, after a restart, after all systems have restarted, run postboot, which does something like:<br />
<br />
for f in `df -k | grep /dev/vn | awk '{print $6}'` ; do chmod 0666 $f/dev/null ; done<br />
<br />
Later, after the system is up, if anyone ever complains about their /dev/null permissions for any reason ... really if anyone mentions<br />
/dev/null in any way, tell them to run:<br />
<br />
chmod 0666 /dev/null<br />
<br />
We have ultimately fixed this by adding chmods to the quad/safe's<br />
<br />
<br />
== mknod in a jail == <br />
<br />
For a customer who wants to run a chroot name server (named) in a jail.<br />
<br />
<pre><br />
cd /mnt/data1/<user directory>/var/named<br />
mknod -c 1 1 null<br />
mknod -c 2 3 random<br />
</pre><br />
<br />
== Postfix Problems ==<br />
<br />
This hasn't happened in a long time, probably no longer valid with modern OS's<br />
<br />
Postfix is an alternate MTA - people replace sendmail with it in much the same way that many people also replace sendmail with qmail. The problem is that if you install postfix inside of a jail, by default it will not work properly - the user will see error messages like this:<br />
<br />
incoming mail generates errors like this:<br />
<br />
mail_queue_enter: create file incoming/298266.55902: File too large<br />
<br />
and further, sending mail generates errors like this:<br />
<br />
postdrop: warning: mail_queue_enter: create file maildrop/627930.56676: File too large<br />
<br />
This is very easy to solve. In fact, the FreeBSD welcome email that is sent out by `jailmake` now contains this block of text:<br />
<br />
- if you plan on installing postfix, email us first and ask for a necessary patch.<br />
<br />
So, anytime you see a support email that says anything about postfix not working, paste that line from the welcome email into your response to them, and right away, forward the postfix patch to them in an email.<br />
<br />
Here are the instructions:<br />
<br />
Simply copy the patch into /usr/ports/mail/postfix/files, rename the file to ‘patch-file_limit.c’ and then cd to /usr/ports/mail/postfix and run `make install`.<br />
<br />
The postfix patch is the very first email in the support email box, and we never delete or save it because we always want it there to forward to people. The instructions in the email are very clear, and it solves their problem 100% of the time.<br />
<br />
<br />
== Problems with `find` ==<br />
<br />
There was a bug in the FreeBSD vn-filesystem code. The bug causes the system to crash or hang when someone runs the `find` command inside their vn-backed filesystem (and even though the bug may have been fixed, due to the i/o overhead, we still don't like find).<br />
<br />
Now, this does not always happen - however, any jail machine with more than 20 systems on it will crash every night if every system on it runs the daily periodic script out of their crontab, and updates their locate database with the find command.<br />
<br />
(the locate database is a small, FreeBSD specific database that is populated nightly from a cron job, and is then used to provide fast answers to the `locate` command)<br />
<br />
In normal systems (32-40 jails on them), all that has to be done is make sure nobody runs the daily periodic. This means that the file /etc/crontab, instead of looking like this:<br />
<br />
<pre># do daily/weekly/monthly maintenance<br />
1 3 * * * root periodic daily<br />
15 4 * * 6 root periodic weekly<br />
30 5 1 * * root periodic monthly</pre><br />
<br />
Needs to look like this:<br />
<br />
<pre># DO NOT UNCOMMENT THESE - contact support@johncompanies.com for details<br />
#1 3 * * * root periodic daily<br />
#15 4 * * 6 root periodic weekly<br />
#30 5 1 * * root periodic monthly<br />
## DO NOT UNCOMMENT THESE ^^^^^^^^</pre><br />
<br />
and the problem will generally not occur at all. New filesystem images are always altered in this way, so all systems are set like this.<br />
<br />
However, if you have enough systems running, even casual uses of find, outside of the periodic scripts, can cause the system to crash. We are seeing this on jail13, which crashes about once per week.<br />
<br />
There is not yet any resolution to this problem.<br />
<br />
So, if you have a freebsd system crash, see if anyone has uncommented their daily periodic line - you can see them for everyone by running:<br />
<br />
/bin/sh<br />
<br />
and then running:<br />
<br />
<pre>for f in `df -k | grep vn | awk '{print $6}'` ; do echo `cat $f/etc/crontab | grep "periodic daily"` $f ; done | more</pre><br />
<br />
(all one line, one command)<br />
<br />
And then edit those /etc/crontabs and comment them out again.<br />
<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
* if you are getting disk full messages for a BSD customer, it's fairly safe to clear out their /usr/ports/distfiles dir<br />
* 4.x: ps and top an only be run by root in these jails. Done on purpose:<br />
As for `ps` and `top` - non-root users can run them, just not successfully because we have locked the permissions on /dev/mem and /dev/kmem to be root-readable only. That is why non-root users cannot successfully run `ps` and `top`.<br />
* user quotas do not work on freebsd jails - you cannot set up quotas at all, and that's that.<br />
* You cannot inject a process into a 4.x jail, only HUP running processes. 6.x and onward you can with jexec<br />
* jails see base machine’s uptime/load when running top/w<br />
* if someone is unable to get in- cant ping, etc- see if they were blocked by castle (DoS), see if their ip is on the system (post reboot it was lost cause wasn’t in the rc.conf). preboot should catch that.<br />
* in FreeBSD you cant su to root unless you belong to wheel group – so if you remove your acct and setup a new one, we have to add it to wheel group (add to /etc/group)<br />
* Dmesg from underlying sys is seen in customers dmesg on jail<br />
* Popper process going crazy @ 40% for 10min = someone who leaves mail on server<br />
* Don’t force umounts on 4.x jails – it crashes the machine, generally ok on newer machines (running md)<br />
* Good book for admin http://search.barnesandnoble.com/booksearch/isbnInquiry.asp?userid=t824VyRAYz&isbn=0596005164&itm=2<br />
* Self-sign ssl cert http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#selfcert<br />
* conversation with Glenn about semaphores and pgsql<br />
<pre>SDBoody: hey, these are valid amounts/figures/increments right:<br />
kern.ipc.semmni=1280<br />
kern.ipc.semmns=1280<br />
<br />
gr8feen: probably... I always forget exactly what those are, so I usually have to look them up<br />
SDBoody: semaphores<br />
SDBoody: i took the current 1024 and added 256 to them<br />
SDBoody: need more for pgsql<br />
gr8feen: I meant the mni and mns parts...<br />
gr8feen: some of those are not ones you just want to add to... hang on a sec and I'll look them up..<br />
gr8feen: what's semmsl set to?<br />
SDBoody: kern.ipc.semmsl: 1024<br />
<br />
SDBoody: kern.ipc.msgseg: 2048 <br />
kern.ipc.msgssz: 8 <br />
kern.ipc.msgtql: 40 <br />
kern.ipc.msgmnb: 2048 <br />
kern.ipc.msgmni: 40 <br />
kern.ipc.msgmax: 16384 <br />
kern.ipc.semaem: 16384 <br />
kern.ipc.semvmx: 65534 <br />
kern.ipc.semusz: 152 <br />
kern.ipc.semume: 10 <br />
kern.ipc.semopm: 100 <br />
kern.ipc.semmsl: 1024 <br />
kern.ipc.semmnu: 512 <br />
kern.ipc.semmns: 1024 <br />
kern.ipc.semmni: 1024 <br />
kern.ipc.semmap: 768 <br />
kern.ipc.shm_allow_removed: 0 <br />
kern.ipc.shm_use_phys: 1 <br />
kern.ipc.shmall: 262144 <br />
kern.ipc.shmseg: 256 <br />
kern.ipc.shmmni: 784 <br />
kern.ipc.shmmin: 1 <br />
kern.ipc.shmmax: 536870912 <br />
kern.ipc.maxsockets: 25600 <br />
<br />
gr8feen: ok...msl is max per id, mni is max ids, mns is max number of semaphores... so you probably want something like mns = mni * msl<br />
gr8feen: which one did you run out of?<br />
SDBoody: not sure how to tell- ipcs shows the sems in use add up to 1024<br />
SDBoody: there are 59 entries<br />
gr8feen: I'm assuming you tried to start postgres and it failed?<br />
SDBoody: yes<br />
gr8feen: it should have logged why, somewhere..<br />
gr8feen: if I recall, it'll tell you which one it ran out of<br />
SDBoody: > DETAIL: Failed system call was semget(1, 17, 03600).<br />
<br />
gr8feen: so it wanted an id with 17 semaphores...I'd start by making mns = 17*mni and leave mni and mnl set to what they are now and see what it does<br />
SDBoody: i think mni is plenty high<br />
SDBoody: ok, more reasonable 17408<br />
gr8feen: yeah...just change that one and see how it goes..<br />
SDBoody: and leave mni alone at 1024?<br />
gr8feen: yeah...mni id the max number of ids...but if it's trying to get something like 17 per id, your going to hit mns before you hit anything else<br />
SDBoody: right, but doesn't hurt to have it that high (assuming)<br />
<br />
gr8feen: not really... I think those get allocated out of ram that you cant page out, but it's still such a small amount that it really doesn't matter<br />
SDBoody: looks like that worked, thx!<br />
gr8feen: cool<br />
<br />
see semaphores: ipcs -a -s</pre><br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
== Making new customer VE (vm) ==<br />
<br />
This applies only to new virts >= 4.x<br />
<br />
grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.<br />
<br />
1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
also, ping the IP address.<br />
<br />
2. put CT on whichever partition has more space<br />
df -h<br />
<br />
3. vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> ) <br />
vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@johncompanies.com centos-6-x86_64 LM<br />
<br />
4. copy veid, dir, ip and password to pending customer screen. activate customer<br />
<br />
== Making new customer VE (vemakexxx) ==<br />
<br />
This applies to older virts with old templates. This should probably not be used at all anymore.<br />
<br />
<br />
1. look thru hist for ip<br />
<br />
2. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
<br />
3. put ve on whichever partition has more space<br />
vemakerh9 <veid> <ip> <hostname> <mount> <email> [gb disk]; <256|384|512> <veid><br />
vemakerh9 866 69.55.226.109 ngentu.com /vz1 ayo@ngantu.com,asd@asd.com 5; 256 866<br />
<br />
4. copy (veid), dir, and ip to pending customer screen (pass set to p455agfa)<br />
<br />
<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrateonline/migrate failures (migrate manually) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
You may also see migration failing due to quota issues.<br />
<br />
You can try to resolve by copying any quota file into the file you need:<br />
<br />
cp /var/vzquota/quota.1 /var/vzquota/quota.xxx<br />
<br />
If it complains about quota running you should then be able to stop it<br />
<br />
vzquota off xxxx<br />
<br />
If all else fails, migrate to a new VEID<br />
i.e. 1234 becomes 12341<br />
<br />
<br />
If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
'''Note:''' don't forget to fix /usr/local/sbin/backup.config for new disk location<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
=== Cant change file /etc/sysconfig/network ===<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
=== What template is user using ===<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
=== Edits needed to run java ===<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
=== Set time zone ===<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
=== Failing shm_open calls ===<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
=== Debian can't get on the network ===<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
=== CentOS can't get on the network ===<br />
<br />
When CentOS system was running, but can't get on the network, they probably updated iproute via yum.<br />
<br />
Copy an older version of the iproute rpm to their VPS.<br />
<br />
<pre>cp -p /root/iproute/iproute-2.6.32-23.el6.<i686>.rpm /vz2/private/<1527>/fs/root</pre><br />
<br />
Enter their ve, install rpm, restart network, and prevent yum from updating iproute.<br />
<br />
<pre><br />
ve <1527><br />
rpm -i --force iproute-2.6.32-23.el6.<i686>.rpm<br />
rpm -q iproute<br />
rpm -e <new iproute rpm><br />
service network restart<br />
echo "exclude=iproute" >>/etc/yum/ip-exclude<br />
echo "include=/etc/yum/ip-exclude" >>/etc/yum.conf<br />
</pre><br />
<br />
<br />
<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
----<br />
<br />
* uptime/load that vz users see isn’t what we see- it's their own load<br />
* Vz overwrites resolve.conf on every boot to what’s in ve config file. Also overwrites hostname<br />
* Quotas in vz: set Gid_limit in vz config file and restart ve. Errors leading up to this will be out of files but no inode or space issues. set to 2000 in conf, and restart vz<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
To create a new Linux VPS<br />
<br />
* get an IP address from mgmt screen<br />
* run df -h to figure out which disk has enough space<br />
* run vm with no arguments to get syntax and exact template name<br />
* use col0xxxx for hostname if they don’t give you a hostname<br />
* copy over dir, veid, ip and password to pending customer screen<br />
<br />
vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> )<br />
<br />
== disable ==<br />
To disable a customers Virtuozzo VPS server.<br />
<br />
* stop the server<br />
vzctl stop <veid><br />
* prevent restart on server reboot and user restart via Control Panel<br />
vzctl set <veid> --disabled=yes --offline_management=no --save<br />
<br />
== cancelve ==<br />
When a customer cancels all service.<br />
<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
<br />
display the contents of the vz config file for the <veid><br />
<br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
<br />
List all virtual environments running on this machine.<br />
<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
<br />
show processes running under this <veid>.<br />
<br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
<br />
Restart a <veid>.<br />
<br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
<br />
enter a <veid> as root.<br />
<br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
<br />
execute a single command in the <veid> environment.<br />
<br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem VEID <256|512|768|1024|1280|1536|2048> [burst GB]<br />
adjusts the memory resources for the VE. If this is on VZ ver >4, you will also be prompted to provide a burst ram amount. Sample usage:<br />
<br />
<pre># setmem 21671 1024 4<br />
/usr/sbin/vzctl set 21671 --ram 1024m --swap 3072m --save<br />
</pre><br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>75.80.128.147https://wiki.jcihosting.com/index.php?title=Infrastructure_Machines&diff=2213Infrastructure Machines2020-03-17T18:25:04Z<p>75.80.128.147: /* virt11 */</p>
<hr />
<div>= jails =<br />
<br />
== jail1 ==<br />
<br />
* Location: castle, SHUTDOWN<br />
* OS: FreeBSD 6.2 i386<br />
* Networking: Priv IP: 10.1.4.101 (PCI nic), Pub IP: 69.55.230.107 (onboard)<br />
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.<br />
* Remote management: none<br />
* Disk accounting: gvinum<br />
<br />
== jail2 ==<br />
<br />
* Location: castle, cab 6-16<br />
* OS: FreeBSD 7.2 amd64<br />
* Networking: Priv IP: 10.1.4.102, Pub IP: 69.55.228.53 (2 onboard nics)<br />
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: one 146 GB (2 x 146GB) RAID1 array, two 300 GB (4 x 300GB) RAID1 arrays running on an LSI-based, Dell-branded (PERC 6/i) RAID card.<br />
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.232 <br />
* Disk accounting: md<br />
<br />
== jail3 ==<br />
* Location: I2b SHUTDOWN<br />
* OS: FreeBSD 8.3 amd64<br />
* Networking: Priv IP: 10.1.2.103, Pub IP: 69.55.229.7 (2 onboard nics)<br />
* Hardware: Supermicro (custom build). 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: two 300 GB (2 x 300GB) RAID1 array running on a 3ware 8006-2LP RAID card.<br />
* Remote management: none<br />
* Disk accounting: md<br />
<br />
=== Notes ===<br />
* '''We should not add users to this server since it is at I2B'''<br />
* must be ssh'd to from nat2<br />
* is a super jail for customer col01737<br />
<br />
== jail4 ==<br />
<br />
* Location: castle, cab 6-17<br />
* OS: FreeBSD 9.1 x86_64<br />
* Networking: Priv IP: 10.1.4.104, Pub IP: 69.55.228.104 (2 onboard nics) <br />
* Hardware: Dell 2850. 6 x 300GB SCSI drives (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.<br />
* CPU: 2 x Intel(R) Xeon(TM) CPU 2.80GHz (8 virtual CPUs)<br />
* RAM: 16 GB ( 4 x 4GB Reg ECC )<br />
* Drives: one 1.4 TB RAID 5 array (6 x 300GB SCSI) Dell-branded (PERC 4e)LSI megarc RAID card.<br />
* Remote management: None <br />
* Disk accounting: md<br />
<br />
<br />
=== Notes ===<br />
Only FreeBSD 9.1 jail<br />
Not upgraded to FBSD 9.2 or 9.3 because too many libraries modified (would require customers to rebuild apps).<br />
<br />
== jail5 ==<br />
<br />
* Location: castle, cab 3-6<br />
* OS: FreeBSD 10.1 x86_64<br />
* Networking: Priv IP: 10.1.4.105, Pub IP: 69.55.230.105 (2 onboard nics) <br />
* Hardware: Supermicro JC-14004 - Intel S1200BTL motherboard - 6 SATA/SAS drive bays (2 colums of 3), Dual power supply.<br />
* CPU: 1 x Intel(R) Xeon(TM) E3-1230 V2 CPU 3.30 GHz (8 virtual CPUs)<br />
* RAM: 32 GB ( 4 x 8GB ECC )<br />
* Drives: 1x80 GB SATA SSD on motherboard + one 2.6 TB RAID 5 array 4x1 TB + 3ware 9650 RAID card.<br />
* Remote management: Intel RMM 4 - 10.1.4.235<br />
* Disk accounting: md<br />
<br />
=== Notes ===<br />
Only FreeBSD 10.1 jail used for bhyve virtuals.<br />
<br />
Use ~+Ctrl-D to disconnect from console (vm attach colXXXXX).<br />
<br />
== jail6 ==<br />
<br />
* Location: castle, cab 6-16<br />
* OS: FreeBSD 10.3 x86_64<br />
* Networking: Priv IP: 10.1.4.106, Pub IP: 69.55.230.106 (2 onboard nics) <br />
* Hardware: Supermicro JC-14004 - Intel S1200BTL motherboard - 6 SATA/SAS drive bays (2 colums of 3), Dual power supply.<br />
* CPU: 1 x Intel(R) Xeon(TM) E3-1230 V2 CPU 3.30 GHz (8 virtual CPUs)<br />
* RAM: 32 GB ( 4 x 8GB ECC )<br />
* Drives: one 2.7 TB ZFS RAID 6 array 5x1 TB <br />
* Remote management: Intel RMM 4 - 10.1.4.236<br />
* Disk accounting: zfs<br />
<br />
== jail7 ==<br />
<br />
* Location: castle, cab 3-5 SCHEUDLED SHUTDOWN 9/30/19<br />
* OS: FreeBSD 6.3 i386<br />
* Networking: Priv IP: 10.1.4.107, Pub IP: 69.55.230.108 (2 onboard nics)<br />
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: two 146 GB (4 x 146GB) RAID1 arrays, one 74 GB (2 x 74GB) RAID1 array running on an LSI-based, Dell-branded (PERC 6/i) RAID card.<br />
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.237<br />
* Disk accounting: gvinum<br />
<br />
=== Notes ===<br />
Do not run a verify while OS/jails running, will crash.<br />
<br />
== jail8 ==<br />
<br />
* Location: castle, cab 3-6<br />
* OS: FreeBSD 8.0 amd64<br />
* Networking: Priv IP: 10.1.4.108, Pub IP: 69.55.234.2 (2 onboard nics)<br />
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 300 GB (2 x 300GB) RAID1 array, one 400 GB (2 x 400GB) RAID1 array, running on an LSI-based, Dell-branded (PERC 6/i) RAID card.<br />
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.238<br />
* Disk accounting: md<br />
<br />
<br />
== jail9 ==<br />
<br />
* Location: castle, cab 3-6<br />
* OS: FreeBSD 8.2 amd64<br />
* Networking: Priv IP: 10.1.4.109, Pub IP: 69.55.232.36 (2 onboard nics)<br />
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 400 GB (2 x 300GB) RAID1 array running on an LSI-based, Dell-branded (PERC 5/i) RAID card.<br />
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.239<br />
* Disk accounting: md<br />
<br />
<br />
== jail11 ==<br />
<br />
* Location: castle, cab 3-7<br />
* OS: FreeBSD 4.7 i386<br />
* Networking: Priv IP: 10.1.4.111 (PCI nic), Pub IP: 69.55.236.92 (onboard)<br />
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.<br />
* Remote management: none<br />
* Disk accounting: vinum<br />
<br />
<br />
== mx1 ==<br />
<br />
* Location: castle, SHUTDOWN AND SCRAPPED<br />
* OS: FreeBSD 4.11 i386<br />
* Networking: Priv IP: 10.1.4.201 (PCI nic), Pub IP: 69.55.237.3 (onboard)<br />
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: one 36 GB (2 x 36GB) RAID1 array, one 74 GB (2 x 74GB) RAID1 array running on an Adaptec-based, Dell-branded (perc) RAID card.<br />
* Remote management: none<br />
* Disk accounting: vinum<br />
<br />
=== Notes ===<br />
* is our (old) backup mail/dns vps service host<br />
<br />
== mx2 ==<br />
<br />
* Location: castle, SHUTDOWN AND SCRAPPED<br />
* OS: FreeBSD 7.1 i386<br />
* Networking: Priv IP: 10.1.4.202 (PCI nic), Pub IP: 69.55.237.90 (onboard)<br />
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.<br />
* Disk accounting: gvinum<br />
<br />
=== Notes ===<br />
* is our latest backup mail/dns vps service host<br />
<br />
== jail17 ==<br />
* Location: castle, cab 3-7<br />
* OS: FreeBSD 4.10 i386<br />
* Networking: Priv IP: 10.1.4.117 (PCI nic), Pub IP: 69.55.228.2 (onboard nics)<br />
* Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: two 74 GB (4 x 74GB) RAID1 arrays, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S RAID card.<br />
* Remote management: none<br />
* Disk accounting: vinum<br />
* Host of devweb.johncompanies.com and www.utopian.com/mail.utopian.com<br />
<br />
== jail18 ==<br />
* Location: castle, cab 3-5 SCHEDULED SHUTDOWN 9/30/19<br />
* OS: FreeBSD 4.10 i386<br />
* Networking: Priv IP: 10.1.4.118 (PCI nic), Pub IP: 69.55.228.2 (onboard nics)<br />
* Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: two 74 GB (4 x 74GB) RAID1 arrays, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S RAID card.<br />
* Remote management: none<br />
* Disk accounting: vinum<br />
* Host of ns2c.johncompanies.com (now on ns2c.johncompanies.com on ganeti)<br />
<br />
== jail19 ==<br />
* Location: castle, cab 3-5 SCHEDULED SHUTDOWN 9/30/19<br />
* OS: FreeBSD 6.1 i386<br />
* Networking: Priv IP: 10.1.4.119 (PCI nic), Pub IP: 69.55.228.200 (onboard nics)<br />
* Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: one 74 GB (2 x 74GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S RAID card.<br />
* Remote management: none<br />
* Disk accounting: gvinum<br />
<br />
= virts =<br />
<br />
== quar1 ==<br />
<br />
* Location: castle, SHUTDOWN AND SCRAPPED<br />
* OS: RedHat 7.3 x86<br />
* Networking: Priv IP: 10.1.4.151 (PCI nic), Pub IP: 69.55.227.2 (onboard nic)<br />
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: one 36 GB (2 x 36GB) RAID1 array, one 74 GB (2 x 74GB) RAID1 array, running on an Adaptec-based, Dell-branded (perc) RAID card.<br />
* Remote management: none<br />
* Virtuozzo version: 2.6.1<br />
* VZ license: hwid=23C0.C0E1.6FDD.08BA.8971.8E1C.EBD5.1EDC serial=0DE6.903E.E239.E23F.470C.4369.4104.A5A4<br />
<br />
=== Notes ===<br />
* used to be the home of customers who's VE's would just run out of control/badly<br />
* has a max of 10 VE's allowed to run<br />
<br />
== virt9 ==<br />
<br />
* Location: castle, cab 3-7<br />
* OS: RedHat 7.3 x86<br />
* Networking: Priv IP: 10.1.4.59 (PCI nic), Pub IP: 69.55.226.161 (onboard nic)<br />
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: two 74 GB (2 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.<br />
* Remote management: none<br />
* Virtuozzo version: 2.6.1<br />
* VZ license: hwid=BC15.B4D6.0D25.A5FE.F3BA.D518.E351.AE3F serial=F6AD.B6B4.5650.8869.C97C.73EE.AF65.FA8B<br />
<br />
<br />
== virt11 ==<br />
<br />
* Location: castle, cab 3-6<br />
* OS: CentOS 5.4 x86<br />
* Networking: Priv IP: 10.1.4.61, Pub IP: 69.55.238.3, 2 onboard nics<br />
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 400 GB (2 x 400GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.<br />
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.211<br />
* Virtuozzo version: 4.0.0<br />
* VZ license: hwid="029D.A187.78E1.480F.49E3.E20A.7389.7F79" serial="163C.F3E2.195F.96B5.2D38.8937.9600.4A05" key_number="VZ.00172378.0006"<br />
<br />
== virt12 ==<br />
<br />
* Location: castle, cab 3-7<br />
* OS: CentOS 5.2 x86<br />
* Networking: Priv IP: 10.1.4.62, Pub IP: 69.55.227.70, 2 onboard nics<br />
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: two 300 GB (2 x 300GB) RAID1 arrays one 400 GB (2 x 400GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.<br />
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.212<br />
* Virtuozzo version: 4.0.0<br />
* VZ license: hwid="0C53.A413.E095.B4F4.51BC.D740.6919.A77B" serial="84E5.9498.3759.E683.E24B.2514.CA72.DC31"<br />
<br />
== virt13 ==<br />
<br />
* Location: castle, cab 6-17<br />
* Switch port: P13-<br />
* OS: CentOS 6.2 x86_64<br />
* Networking: Priv IP: 10.1.4.63, Pub IP: 69.55.226.2, 2 onboard nics<br />
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* CPU: 2 x Intel(R) Xeon(R) CPU E5420 @ 2.50GHz (8 virtual cores)<br />
* RAM: 32 GB (8 x 4GB DDR2 FB-DIMM ECC 667MHz)<br />
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 600 GB (2 x 600GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.<br />
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.213<br />
* Virtuozzo version: 4.7.0<br />
* VZ license: hwid="7D07.93BE.0B1F.7D2B.B039.4B5B.48B6.453B" serial="60A4.A94C.44BB.DCD6.8D03.1778.605B.10FE"<br />
<br />
=== Notes ===<br />
* home to our latest/current signups<br />
* currently the only 64bit vz host<br />
<br />
== virt14 ==<br />
<br />
* Location: castle, cab 6-16<br />
* Switch Port: p13-<br />
* OS: CentOS 6.4 x86_64<br />
* Networking: Priv IP: 10.1.4.64 Pub IP: 69.55.225.14 2 onboard nics<br />
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.<br />
* CPU: 2 x Xeon 5140 Dual Core @ 2.33GHz (4 virtual CPUs)<br />
* RAM: 32 GB (8 x 4GB Reg ECC)<br />
* Drives: one 146 GB (2 x 146 GB SAS) RAID1 array, and one 1TB RAID1 array (2 x 1 TB SATA), running on an LSI-based, Dell-branded (perc 5/i) RAID card.<br />
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.214<br />
* Virtuozzo version: 4.7.0<br />
* VZ license: hwid="" serial=""<br />
<br />
=== Notes ===<br />
* our latest virt <br />
* Temp server to offload Virt13 till we can get a Cloud going.<br />
* virt 13 and 14 currently the only 64bit vz hosts<br />
<br />
== virt15 ==<br />
<br />
* Location: SHUTDOWN<br />
* OS: RedHat 9 x86<br />
* Networking: Priv IP: 10.1.4.65, Pub IP: 69.55.232.160 (2 onboard nics)<br />
* Hardware: Supermicro (custom build). 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: one 74 GB (2 x 74GB) RAID1 array, two 146 GB (2 x 146GB) RAID1 arrays, running on an LSI MegaRAID SCSI 320-1 RAID card.<br />
* Remote management: none<br />
* Virtuozzo version: 2.6.2<br />
* VZ license: hwid=A90F.6F48.E723.D8BA.3025.184A.5B73.D11E serial=E94B.5164.C1E6.A67F.67D1.7D96.0B6C.5524<br />
<br />
== virt16 ==<br />
<br />
* Location: castle, cab 3-7<br />
* OS: Fedora Core 4 x86<br />
* Networking: Priv IP: 10.1.4.66, Pub IP: 69.55.232.2 (2 onboard nics)<br />
* Hardware: Supermicro (custom build). 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: one 74 GB (2 x 74GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an LSI MegaRAID SCSI 320-1 RAID card.<br />
* Remote management: none<br />
* Virtuozzo version: 3.0.0<br />
* VZ license: hwid=DEFA.A325.7230.BBC8.9715.8B52.3FD7.27BE serial=66C0.41EA.3FBB.11D3.9CC6.55C7.09AE.14AB<br />
<br />
<br />
== virt17 ==<br />
<br />
* Location: castle, cab 3-6<br />
* OS: CentOS 4.4 x86<br />
* Networking: Priv IP: 10.1.4.67, Pub IP: 69.55.232.162, 2 onboard nics<br />
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: two 146 GB (2 x 146GB) RAID1 arrays running on an LSI-based, Dell-branded (perc 5/i) RAID card.<br />
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.217<br />
* Virtuozzo version: 3.0.0<br />
* VZ license: hwid=2E14.AED9.70B8.C26E.D99F.B0D3.BCD2.229C serial=2A11.DAD0.61DB.E889.8DF4.9AF7.CF82.3C37<br />
<br />
<br />
== virt19 ==<br />
<br />
* Location: castle, cab 3-6<br />
* OS: CentOS 5.2 x86<br />
* Networking: Priv IP: 10.1.4.69, Pub IP: 69.55.236.2, 2 onboard nics<br />
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 300 GB (2 x 300GB) RAID1 array, running on an LSI-based, Dell-branded (perc 5/i) RAID card.<br />
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.219<br />
* Virtuozzo version: 3.0.0<br />
* VZ license: hwid=3968.13F7.B2AC.8952.8E19.13A9.6EF5.5822 serial=061D.84CD.CCE5.B213.15B5.C061.D6A7.B034<br />
<br />
= mail =<br />
== Summary ==<br />
This machine (mail) is the swiss army knife of the company, playing host to many services and functions. <br />
<br />
* Location: castle, cab 3-7<br />
* OS: FreeBSD 4.10 x86<br />
* Networking: Priv IP: 10.1.4.5, Pub IPs: 69.55.230.2, 69.55.225.225 (ns1c jail), 69.55.230.9. 1 onboard and 1 PCI<br />
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.<br />
<br />
== Services Provided ==<br />
* mail<br />
* web<br />
* mysql<br />
* bigbrother server/pager<br />
* snmp<br />
* named in a jail (ns1c)<br />
<br />
== email ==<br />
This server hosts mail for johncompanies.com (mail.johncompanies.com). Sendmail 8.13.6/8.13.6 is listening on 69.55.230.2 port 25 for incoming mail. Relaying is allowed per /etc/mail/relay-domains<br />
Other addresses (aliases) are defined per /etc/mail/aliases<br />
<br />
The following active users have mail hosted on this server:<br />
* dave<br />
* linux<br />
* support<br />
* payments<br />
* sales<br />
* tech1<br />
* info<br />
<br />
Traditionally, mail is checked via shell apps (pine). qpopper (pop3s) is running to allow mail downloading. Checking mail in this way causes an opened INBOX in pine to lock read-only. For this reason, we tee incoming mail to support and linux to tech1.<br />
<br />
Procmail rules are setup to filter spam and send text messages. They are enabled for info, support, linux, tech1, dave and can be found in ~/Procmail/, for example:<br />
<pre># more ~support/Procmail/rc.emergency<br />
:0c # use c only if you want to forward a copy and file the original later<br />
* ^Subject:.*\<emergency\><br />
* ! ^Subject:\<re\><br />
{<br />
:0h<br />
FROMANDSUBJECT=|formail -XFrom: -XSubject:<br />
<br />
:0fwh<br />
| /usr/local/bin/formail -I"Subject: " -I"To: pager@johncompanies.com" ; echo $FROMANDSUBJECT ; echo<br />
<br />
:0<br />
! -t<br />
}<br />
<br />
</pre><br />
<br />
control: <tt>cd /etc/mail; make stop</tt> (stop), <tt>cd /etc/mail; make start</tt> (start)<br />
<br />
The following aliases are also in place:<br />
<br />
<pre>debian: linux<br />
jobs: info<br />
careers: info<br />
#reboot: 6128102202@txt.att.net<br />
#reboot: 8582298897@vtext.com<br />
reboot: pager<br />
#pager: 8582298897@vtext.com<br />
pager: 4158718324@txt.att.net<br />
tech1on: "| /usr/local/sbin/tech1on.sh"<br />
tech1off: "| /usr/local/sbin/tech1off.sh"</pre><br />
<br />
To change them, edit <tt>/etc/aliases</tt> and then run <tt>newaliases</tt><br />
<br />
Note on tech1: this address was setup as a read-only address to be mirrored on all email coming into support and linux. We set this up so we could easily check support mail via a pop client- popping email locks out the user in pine so checking support/linux directly via pop was not an option. When checking and responding to email that comes into tech1, care should be taken to make sure it is sent as/under an address other than tech1. This is cause tech1 is not monitored by support staff as closely as email to support/linux. Further, the tech on call may not be checking tech1. Lastly, because of the nature of the copying, you will sometimes notice certain automated email/notices are received 2x in support- this is because of/related to the tech1 mirror.<br />
<br />
To enable it (on mail, run):<br />
<br />
<tt>~support/tech1on.sh</tt><br />
<br />
To disable<br />
<br />
<tt>~support/tech1off.sh</tt><br />
<br />
Or via email:<br />
<br />
<tt>tech1on@johncompanies.com<br />
tech1off@johncompanies.com</tt><br />
<br />
== IP Blocking ==<br />
<br />
<pre><br />
01000 deny ip from 188.92.72.5 to any<br />
01003 deny ip from any to 122.49.31.50<br />
01004 deny ip from 122.49.31.50 to any<br />
01014 deny ip from 74.208.225.225 to any<br />
01015 deny ip from any to 216.243.118.35<br />
01016 deny ip from 216.243.118.35 to any<br />
01017 deny ip from any to 216.243.118.36<br />
01018 deny ip from 216.243.118.36 to any<br />
01020 deny ip from 112.215.0.0/18 to any 2014-08-13 Blocked PT Excelcomindo Pratama (Indonesia) for fradulent credit card attempts <br />
01020 deny ip from 112.215.64.0/20 to any 2014-08-13 Blocked PT Excelcomindo Pratama (Indonesia) for fradulent credit card attempts <br />
01022 deny ip from 120.168.0.0/24 to any 2014-08-13 Blocked Indosat 3G Broadband (Indonesia) for fradulent credit card attempts<br />
01022 deny ip from 120.175.213.0/24 to any 2014-08-13 Blocked Indosat 3G Broadband (Indonesia) for fradulent credit card attempts<br />
<br />
</pre><br />
<br />
== web == <br />
<br />
See [[Management_System_/_Public_Website_/_Signup|Management System / Public Website / Signup]]<br />
<br />
== mysql ==<br />
mysql 4.1.22 is running on port 3306<br />
<br />
* datadir: <tt>/mnt/data1/db/mysql/</tt><br />
* config: <tt>/etc/my.cnf</tt><br />
* database: <tt>jc</tt><br />
* control: <tt>/usr/local/etc/rc.d/mysql-server.sh stop</tt> (stop), <tt>/usr/local/etc/rc.d/mysql-server.sh start</tt> (start)<br />
<br />
== bigbrother ==<br />
There is a client running on mail (which monitors the services running on mail and mail itself), installed under <tt>/usr/home/bb/bbc1.9e-btf</tt><br><br />
And the big brother pager/server (which displays information gathered from all bb-monitored machines, including mail) is installed under <tt>/usr/home/bb/bbsrc/bb1.9i-btf</tt><br />
<br />
Both are running under the user <tt>bb</tt><br />
<br />
Refer to [[BigBrother]] for more about use.<br />
<br />
== DNS (ns1c.johncompanies.com) ==<br />
ns1c is a jail running on the mail server, who's IP is 69.55.225.225<br />
<br />
It's running from <tt>/mnt/data1/ns1c-dir</tt><br />
<br />
See [[DNS]] for more details<br />
<br />
== Usage and Notes ==<br />
* always mounted to backup1 and backup2 via nfs: <br />
<pre>backup2:/mnt/data1 on /backup (nfs)<br />
backup2:/mnt/data2 on /backup2 (nfs)<br />
backup2:/mnt/data3 on /backup3 (nfs)<br />
backup2:/mnt/data4 on /backup4 (nfs)<br />
backup1:/data on /backup1 (nfs)<br />
</pre><br />
<br />
== Cronjobs ==<br />
* * * * * /usr/local/www/mgmt/mrtg/mrtg.sh > /dev/null 2>&1<br />
Gathers up data for our mrtg/load graphs<br />
<br />
*/5 * * * * /usr/local/bin/rsync -a root@nat2:/mnt/data1/mrtg/data/ /usr/local/www/mgmt/mrtg/data/<br />
Gathers up data from i2b servers for our mrtg/load graphs<br />
<br />
40 0 * * * /usr/local/bin/rsync -a root@nat2:"/mnt/data1/mrtg/*.cfg" /usr/local/www/mgmt/mrtg<br />
Gathers up mrtg configuration (port names) from i2b switches for our mrtg/load graphs<br />
<br />
41 0 * * * for f in `grep -l "mnt\/data1" /usr/local/www/mgmt/mrtg/switch-p*.cfg`; do cat $f | sed s#\/mnt\/data1#\/usr\/local\/www\/mgmt# > $f.new; mv $f.new $f; done<br />
Gathers up mrtg configuration (port names) from castle switches for our mrtg/load graphs<br />
<br />
1 0 1 * * cp /usr/local/www/mgmt/html/top20ip /usr/local/www/mgmt/html/top20ip_last<br />
1 0 1 * * cp /usr/local/www/mgmt/html/top20customers /usr/local/www/mgmt/html/top20customers_last<br />
2 * * * * /usr/local/www/cronjobs/top20ip.pl > /dev/null 2>&1<br />
15 * * * * /usr/local/www/cronjobs/top20customer.pl > /dev/null 2>&1<br />
1 0 1 * * rm /usr/local/www/mgmt/html/bandtrack<br />
Archiving and generation of bandwidth statistics presented in mgmt -> Reference -> Bandwidth<br />
<br />
1 0 * * * /usr/local/etc/rsync.backup<br />
Nightly backup script<br />
<br />
0 1 * * * /usr/local/www/mgmt/awstats/wwwroot/cgi-bin/awstats.pl -config=jcpub -update<br />
Public web traffic stats<br />
<br />
15 0 * * * rm /usr/local/www/mgmt/bwgraphs/*.png<br />
16 0 * * * rm /usr/local/www/am/bwgraphs/*<br />
Cleanup for graph-related temp data generated by customers using the bandwidth reports via the AM <br />
<br />
10 0 1 * * /usr/local/www/cronjobs/monthly_bandwidth_report.pl<br />
Monthly bandwidth overage report<br />
<br />
*/3 * * * * /usr/local/www/cronjobs/bbcheck.pl<br />
Updates mgmt with bb monitoring issues<br />
<br />
5 0 * * * /usr/local/www/cronjobs/shutdownreminder.pl<br />
Emails customers reminding them of upcoming shutdown date<br />
<br />
7 0 * * * /usr/local/www/cronjobs/invoice_email.pl<br />
Emails customers who have invoices and are set to auto-email (currently no customer gets these)<br />
<br />
8 */4 * * * /usr/local/www/cronjobs/mysqlrepchk.pl<br />
Checking that we are properly replicating (mysql) traffic data from bwdb to backup1<br />
<br />
16 0 1 * * /usr/local/www/cronjobs/purge_traffic.pl<br />
Removed old traffic data from the traffic database (running on backup1)<br />
<br />
*/5 * * * * chmod 0700 /usr/local/www/ccard_orders/* && mv /usr/local/www/ccard_orders/* /usr/local/www/ccard_orders/done<br />
Secure credit card data: set root-read-only <br />
<br />
25 0 * * * /usr/local/www/cronjobs/biller.pl<br />
Enters service charges in customer billing ledgers<br />
<br />
10 13 * * * /usr/local/www/cronjobs/pfp_batch_gather.pl<br />
Looks for customers with balance due and active credit card on file, prepares a payflow batch<br />
<br />
10 14 * * * /usr/local/www/cronjobs/pfp_batch_process.pl<br />
Tries to collect ccard funds for items in payflow batch - communicates with payflow<br />
<br />
15 13 * * * /usr/local/www/cronjobs/pb_batch_gather.pl<br />
Looks for customers with balance due and active paypal billing agreement on file, prepares a paypal batch<br />
<br />
15 14 * * * /usr/local/www/cronjobs/pb_batch_process.pl<br />
Tries to collect paypal funds for items in paypal batch - communicates with paypal<br />
<br />
0 7 * * 1 /usr/local/www/cronjobs/email_pmt_reminder.pl<br />
Emails customers in arrears, reminding them to pay<br />
<br />
0 0 1 * * /usr/bin/mail -s 'archive sent mail in pine' support@johncompanies.com < /dev/null<br />
Reminds us to archive sent mail<br />
<br />
0 3 * * * /usr/local/bin/rsync -a isys.e-monitoring.net:/var/mail /backup2/isys; /usr/local/bin/rsync -a isys.e-monitoring.net:/usr/home /backup2/isys<br />
Backup data on isys<br />
<br />
== Regular maintenance ==<br />
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]<br />
<br />
== Building a new Mail Server ==<br />
<br />
=== Installations ===<br />
<br />
I used FreeBSD 11.2 <br />
<br />
The order is important especially for the Web Server.<br />
<br />
==== Web Server ====<br />
<br />
I used FreeBSD 11.2<br />
perl 5.26 <br />
OpenSSL 1.0.2o-freebsd<br />
pcre<br />
apache22 <br />
mod_perl2<br />
PayflowPro<br />
mariadb 55 server and client<br />
<br />
Installation order is important<br />
<br />
install perl 5.26.2 from ports<br />
<br />
<ore><br />
cd /usr/ports/lang/perl5.26/<br />
make <br />
[X] PERL_64BITINT Use 64 bit integers (on i386)<br />
[X] USE_PERL Rewrite links in /usr/bin<br />
(the rest unchecked<br />
make install<br />
</pre><br />
<br />
install OpenSSL 1.0.2o-freebsd<br />
<br />
<pre><br />
cd /usr/ports/<br />
make install<br />
</pre><br />
<br />
install pcre<br />
<br />
<pre><br />
cd /usr/ports/<br />
make install<br />
</pre><br />
<br />
<br />
install Apache22<br />
<br />
<pre><br />
cd /usr/ports/distfiles<br />
fetch http://archive.apache.org/dist/httpd/httpd-2.2.32.tar.gz<br />
<br />
<br />
cd /usr/ports/www/apache22/tmp<br />
fetch --no-verify-peer http://mirror.nexcess.net/apache//httpd/httpd-2.2.34.tar.gz<br />
tar xvzf httpd-2.2.34.tar.gz<br />
./configure --prefix=/usr/local/apache --with-ssl=/usr/local/openssl/ --enable-ssl --enable-so --with-mpm=prefork --enable-threads --enable-mods-shared='mime alias setenvif dir' --enable-modules='mime alias setenvif dir' --with-pcre=/usr/local<br />
make install<br />
apachectl restart<br />
<br />
cd /usr/ports/www/apache22<br />
echo "DEFAULT_VERSIONS+=apache=2.2" >> /etc/make.conf<br />
make DISABLE_VULNERABILITIES=yes<br />
make install<br />
</pre><br />
<br />
<br />
install mod_perl2<br />
<br />
<pre><br />
cd /usr/ports/www/mod_perl2<br />
echo "DEFAULT_VERSIONS+=apache=2.2" >> /etc/make.conf<br />
make DISABLE_VULNERABILITIES=yes<br />
make install<br />
</pre><br />
<br />
<br />
install mariadb<br />
<br />
<pre><br />
cd /usr/ports/databases/mariadb-103-server<br />
echo "DEFAULT_VERSIONS+=apache=2.2" >> /etc/make.conf<br />
make DISABLE_VULNERABILITIES=yes<br />
make install<br />
<br />
<br />
</pre><br />
<br />
==== Mail Server ====<br />
<br />
I used Postfix for email<br />
<br />
==== DNS Server (ns1c.johncompanies.com) ====<br />
<br />
= ns2c =<br />
== Summary ==<br />
<br />
* Location: castle, on lamphost ganeti cloud<br />
* OS: FreeBSD 11.2 x86_64<br />
* Networking: Pub IP: 69.55.230.3 Private access: gnt-instance console ns2c.johncompanies.com<br />
* Hardware: on ganeti cloud gn6.jcihosting.net secondary gn1.jcihosting.net<br />
* CPU: 1<br />
* RAM: 1 GB<br />
* Drives: 10 GB<br />
* Remote management: gnt-instance console ns2c.johncompanies.com<br />
<br />
= nat =<br />
== Summary ==<br />
This is the main machine to which we ssh and runs all our screen sessions. Further, it's ip runs in a special block which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.<br />
<br />
* Location: castle, cab 3-7<br />
* OS: FreeBSD 9.1 i386<br />
* Networking: Priv IP: 10.1.4.1, Pub IPs: 69.55.233.195, 69.55.233.196, 69.55.233.197, 69.55.233.198, 69.55.233.199. 1 onboard and 1 PCI<br />
* Hardware: Custom 1U. single power supply. <br />
* Drives: one 8 GB IDE drive<br />
<br />
== Services Provided ==<br />
* nat<br />
<br />
== nat control ==<br />
All rules are contained in and look like:<br />
<br />
<pre>cat /etc/ipnat.rules<br />
# www (was 69.55.230.12)<br />
# virt19<br />
#bimap fxp0 10.1.4.209/32 -> 69.55.233.198/32<br />
# virt18<br />
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32<br />
# virt13<br />
#bimap fxp0 10.1.4.213/32 -> 69.55.233.196/32<br />
# virt12<br />
#bimap fxp0 10.1.4.212/32 -> 69.55.233.196/32<br />
# virt17<br />
bimap fxp0 10.1.4.217/32 -> 69.55.233.196/32<br />
# virt11<br />
#bimap fxp0 10.1.4.211/32 -> 69.55.233.196/32<br />
# ASA<br />
#bimap fxp0 10.1.4.172/32 -> 69.55.233.196/32<br />
# P1A<br />
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32<br />
#bimap fxp0 10.1.4.238/32 -> 69.55.233.197/32<br />
# developer (was 69.55.230.17)<br />
# jail2<br />
#bimap fxp0 10.1.4.232/32 -> 69.55.233.198/32<br />
# jail8<br />
#bimap fxp0 10.1.4.238/32 -> 69.55.233.198/32<br />
# jail9<br />
#bimap fxp0 10.1.4.239/32 -> 69.55.233.198/32<br />
# POLL<br />
#BIMAP EM0 10.1.6.134/32 -> 69.55.230.20/32<br />
# 1U SUN<br />
#BIMAP EM0 10.1.4.4/32 -> 69.55.227.46/32<br />
# ??<br />
#BIMAP EM0 10.1.6.3/32 -> 69.55.230.100/32<br />
# random machine<br />
#bimap fxp0 10.1.6.13/32 -> 69.55.233.199/32<br />
#bimap fxp0 10.1.4.232/32 -> 69.55.233.199/32<br />
# OFFICE OUTBOUND TRAFFIC<br />
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp<br />
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32</pre><br />
<br />
A simple entry looks like:<br />
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32<br />
Which essentially means make private IP 10.1.4.240 reachable on 69.55.233.197 and allow 10.1.4.240 to communicate with the public internet via 69.55.233.197<br />
<br />
To reload new rule config:<br />
ipnat -C -F -f /etc/ipnat.rules<br />
<br />
You may want to setup natting, as above, when you need to reach a DRAC card's web interface, wherin the DRAC card only has a private IP.<br />
<br />
= nat2 =<br />
== Summary ==<br />
This is the main machine to which we ssh and runs all our screen sessions at i2b, and runs ns3c (this is kind of the what mail is to castle). Further, it's ip runs in IP space provided by i2b: 66.181.18.1 - 66.181.18.30, which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.<br />
<br />
* Location: i2b, cab 6<br />
* OS: FreeBSD 6.4 x86<br />
* Networking: Priv IP: 10.1.2.1, Pub IPs: 69.55.229.2, 69.55.229.3, 66.181.18.4, 66.181.18.5, 66.181.18.6, 66.181.18.7, 66.181.18.8, 66.181.18.9, 66.181.18.10, 66.181.18.11, 66.181.18.12, 66.181.18.13, 66.181.18.14 1 onboard and 1 PCI<br />
* Hardware: Custom 2U. 6 drive bays, non-hot-swappable. single power supply. <br />
* Drives: one 150 GB (2 x 150GB) RAID1 array running on a 3ware 8006 RAID card.<br />
<br />
== Services Provided ==<br />
* nat<br />
* bigbrother<br />
* ns3c (jail)<br />
* ntp<br />
<br />
== nat config ==<br />
Here's what's currently nat'd on nat2:<br />
<pre>cat /etc/ipnat.rules<br />
# sample entry<br />
#ATS-9<br />
bimap em0 10.1.2.79/32 -> 66.181.18.14/32<br />
#ATS-8<br />
bimap em0 10.1.2.78/32 -> 66.181.18.13/32<br />
#ATS-7<br />
bimap em0 10.1.2.77/32 -> 66.181.18.12/32<br />
#ATS-6<br />
bimap em0 10.1.2.76/32 -> 66.181.18.6/32<br />
#ATS-5<br />
bimap em0 10.1.2.75/32 -> 66.181.18.7/32<br />
#ATS-4<br />
bimap em0 10.1.2.74/32 -> 66.181.18.8/32<br />
#ATS-3<br />
bimap em0 10.1.2.73/32 -> 66.181.18.9/32<br />
#ATS-2<br />
bimap em0 10.1.2.72/32 -> 66.181.18.10/32<br />
#ATS-1<br />
bimap em0 10.1.2.71/32 -> 66.181.18.11/32<br />
#bwdb2<br />
bimap em0 10.1.2.4/32 -> 66.181.18.5/32<br />
<br />
# spare<br />
<br />
map em0 10.1.2.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp<br />
<br />
#bimap fxp0 10.1.6.49/32 -> 10.1.1.2/32<br />
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32<br />
</pre><br />
<br />
== build ==<br />
<br />
* partition map:<br />
<pre>/ 512m<br />
swap 1G<br />
/var 256m<br />
/tmp 256m<br />
/usr 5g<br />
/mnt/data1 ~</pre><br />
<br />
* edit /etc/make.conf <br />
<pre>echo "WITHOUT_X11=yes \<br />
KERNCONF=nat2 \<br />
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre><br />
<br />
* add settings to /boot/loader.conf and /boot.config<br />
<br />
<pre>echo "-Dh" >> /boot.config<br />
<br />
echo 'console="comconsole,vidconsole" \<br />
boot_multicons="YES" \<br />
boot_serial="YES" \<br />
comconsole_speed="115200"' >> /boot/loader.conf</pre><br />
<br />
* turn off all ttyv's except 0 and 1 in /etc/ttys<br />
also turn on ttyd0, change type to vt100:<br />
<pre>vi /etc/ttys<br />
<br />
ttyv2 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv3 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv4 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv5 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv6 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv7 "/usr/libexec/getty Pc" cons25 off secure<br />
# Serial terminals<br />
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.<br />
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure<br />
<br />
kill -1 1</pre><br />
<br />
on console server:<br />
vi /etc/remote<br />
(rename port to jail8 depending on where and which digi plugged into)<br />
test serial console<br />
<br />
<br />
* populate hosts<br />
<pre>echo "69.55.230.10 backup2" >> /etc/hosts<br />
echo "69.55.230.11 backup1" >> /etc/hosts<br />
echo "10.1.2.4 bwdb2" >> /etc/hosts<br />
echo "10.1.2.3 backup3" >> /etc/hosts</pre><br />
<br />
* put key in authorized_keys on backup3<br />
<pre>cd<br />
ssh-keygen -t dsa -b 1024</pre> <br />
(default location, leave password blank)<br />
<br />
<pre>cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys' <br />
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' <br />
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre> <br />
<br />
confirm that you can ssh to backup3 and backup 2 without getting a login prompt<br />
<br />
ssh backup3 hostname<br />
<br />
ssh backup2 hostname<br />
<br />
ssh backup1 hostname<br />
<br />
<br />
* edit root's path and login script:<br />
vi /root/.cshrc<br />
<br />
Change alias entries (add G):<br />
<pre>alias la ls -aG<br />
alias lf ls -FAG<br />
alias ll ls -lAG<br />
alias ls ls -AG<br />
alias mbm mb mount<br />
alias mbu mb umount</pre><br />
<br />
and alter the prompt, set the following:<br />
set prompt = "`/bin/hostname -s` %/# "<br />
<br />
* install cvsup<br />
<pre>cd /usr/ports/net/cvsup-without-gui <br />
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null</pre><br />
<br />
* get latest sources for this release:<br />
<pre>cd /usr/src <br />
echo "*default host=cvsup4.freebsd.org\<br />
*default base=/usr\<br />
*default prefix=/usr\<br />
*default release=cvs tag=RELENG_6_4\<br />
*default delete use-rel-suffix\<br />
*default compress\<br />
src-all" > sup<br />
<br />
cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null</pre><br />
<br />
<br />
* configure new kernel. <br />
<br />
cd /usr/src/sys/i386/conf <br />
scp backup2:/mnt/data4/build/freebsd/nat2-6.4 ./nat2<br />
<br />
* build, install kernel and world<br />
<br />
<pre>cd /boot<br />
<br />
mv kernel kernel.GENERIC<br />
cd kernel.GENERIC<br />
cd /usr/src<br />
make buildkernel installkernel<br />
<br />
make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null<br />
make installworld <br />
mergemaster -i<br />
</pre><br />
<br />
* populate /etc/rc.conf with IPs and NFS settings<br />
<pre>vi /etc/rc.conf<br />
<br />
hostname="nat2.johncompanies.com"<br />
kern_securelevel_enable="NO"<br />
portmap_enable="NO"<br />
sendmail_enable="NO"<br />
usbd_enable="YES"<br />
<br />
xntpd_enable="YES"<br />
xntpd_flags="-A -p /var/run/ntpd.pid"<br />
<br />
nfs_client_enable="YES"<br />
nfs_reserved_port_only="YES"<br />
ifconfig_em0="inet 10.1.6.50 netmask 255.255.255.0"<br />
#ifconfig_em0="inet 69.55.229.2 netmask 255.255.255.0"<br />
#ifconfig_em0_alias0="inet 69.55.229.229 netmask 255.255.255.255"<br />
ifconfig_fxp0="inet 69.55.229.2 netmask 255.255.255.0"<br />
ifconfig_fxp0_alias0="inet 69.55.229.3 netmask 255.255.255.255"<br />
ifconfig_fxp1="inet 10.1.2.1 netmask 255.255.255.0"<br />
defaultrouter="10.1.6.1"<br />
#defaultrouter=" 66.181.14.250"<br />
snmpd_enable="YES"<br />
ipnat_enable="YES"<br />
ipnat_rules="/etc/ipnat.rules"<br />
gateway_enable="YES"<br />
<br />
inetd_enable="YES"<br />
inetd_flags="-wW -a 10.1.2.1"<br />
fsck_y_enable="YES"<br />
background_fsck="NO"<br />
sshd_enable="YES"</pre><br />
<br />
* reboot. Confirm new kernel is loaded<br />
<br />
uname -a<br />
<br />
* update ports:<br />
<pre>cd /usr/ports<br />
echo "*default host=cvsup4.FreeBSD.org\<br />
*default base=/usr\<br />
*default prefix=/usr\<br />
*default release=cvs tag=RELENG_6_4\<br />
*default delete use-rel-suffix\<br />
*default compress\<br />
ports-all tag=." > sup<br />
<br />
cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null</pre><br />
<br />
* Install raid mgmt tool<br />
<br />
<pre>cd /usr/local/sbin<br />
fetch http://3ware.com/download/Escalade9690SA-Series/9.5.3/tw_cli-freebsd-x86-9.5.3.tgz<br />
tar xzf tw_cli-freebsd-x86-9.5.3.tgz<br />
rm tw_cli-freebsd-x86-9.5.3.tgz<br />
chmod 0700 tw_cli</pre><br />
<br />
Test:<br />
./tw_cli info c0<br />
<br />
* install rsync from ports<br />
<pre>cd /usr/ports/net/rsync<br />
make install clean</pre><br />
choose default options<br />
<br />
* install perl from ports<br />
<pre>cd /usr/ports/lang/perl5.8<br />
make install clean</pre><br />
<br />
* install screen from ports<br />
<pre>cd /usr/ports/sysutils/screen<br />
make install clean</pre><br />
<br />
* install bb client<br />
<pre>adduser<br />
Username: bb<br />
Full name: bb<br />
Uid (Leave empty for default): 1984<br />
Login group [bb]:<br />
Login group is bb. Invite bb into other groups? []:<br />
Login class [default]:<br />
Shell (sh csh tcsh nologin) [sh]: <br />
Home directory [/home/bb]:<br />
Use password-based authentication? [yes]:<br />
Use an empty password? (yes/no) [no]:<br />
Use a random password? (yes/no) [no]: yes<br />
Lock out the account after creation? [no]:<br />
Username : bb<br />
Password : <random><br />
Full Name : bb<br />
Uid : 1984<br />
Class :<br />
Groups : bb<br />
Home : /home/bb<br />
Shell : /bin/sh<br />
Locked : no<br />
OK? (yes/no): yes<br />
<br />
cd /usr/home/bb<br />
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .<br />
tar xvf bb-freebsd.tar</pre><br />
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:<br />
<pre>echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \<br />
10.1.2.1 nat2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts<br />
vi /home/bb/bbc1.9e-btf/ext/openfiles <br />
MACHINE="nat2,johncompanies,com" # HAS TO BE IN A,B,C FORM<br />
<br />
cd /usr/home/bb/bbc1.9e-btf/etc<br />
./bbchkcfg.sh <br />
(y to questions)<br />
./bbchkhosts.sh<br />
(ignore ssh errors)<br />
cd ../..<br />
chown -R bb .<br />
su bb<br />
cd<br />
cd bbc1.9e-btf/src<br />
make; make install<br />
cd ..<br />
<br />
vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh<br />
$1 $TOPARGS > $BBTMP/TOP.$$<br />
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$<br />
<br />
./runbb.sh start<br />
more BBOUT <br />
(look for errors)<br />
exit<br />
<br />
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh<br />
chmod +x /usr/local/etc/rc.d/bb.sh<br />
</pre><br />
<br />
Punch a hole in the firewall to allow it to communicate with bb monitor (probably already exists):<br />
<br />
ipfw add 96 allow ip from 66.181.18.0/27 to 69.55.230.2<br />
<br />
* configure bb on mail:<br />
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts<br />
64.163.14.48 nat2.johncompanies.com # ssh<br />
<br />
su bb<br />
cd<br />
bbsrc/bb/runbb.sh restart ; exit</pre><br />
<br />
* configure ntp<br />
<pre>echo "server 69.55.230.2<br />
server 0.pool.ntp.org<br />
server 1.pool.ntp.org<br />
server 2.pool.ntp.org<br />
server 3.pool.ntp.org<br />
server 66.187.233.4<br />
server 217.204.76.170<br />
server 64.112.189.11<br />
server 66.69.112.130<br />
server 80.85.129.25<br />
server 80.237.234.15<br />
server 130.60.7.44<br />
server 134.99.176.3<br />
server 198.144.202.250<br />
server 202.74.170.194<br />
server 204.17.42.199<br />
server 204.87.183.6<br />
server 213.15.3.1<br />
server 213.239.178.33<br />
server 217.114.97.97<br />
server 69.55.230.2" > /etc/ntp.conf</pre><br />
<br />
<pre>/usr/sbin/ntpd -A -p /var/run/ntpd.pid <br />
sleep 2; ntpq -p</pre><br />
(confirm it’s able to reach our time server)<br />
<br />
echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh<br />
chmod 0700 /usr/local/etc/rc.d/ntp.sh<br />
<br />
* fwd and reverse lookups on ns1c<br />
vr johncompanies.com<br />
(edit the PTR too)<br />
<br />
* setup backups, nfs mount<br />
<br />
<pre>mkdir /backup3<br />
echo 'backup3:/data /backup3 nfs rw,bg 0 0' >> /etc/fstab<br />
<br />
echo '#\!/bin/sh\<br />
backupdir=/data/nat2/current\<br />
\<br />
## ENTRY /etc ' > /usr/local/etc/backup.config</pre><br />
<br />
on backup3:<br />
setup backup dirs:<br />
ssh backup3 mkdir -p /data/nat2/current<br />
<br />
on backup3, add the system to <br />
vi /usr/local/sbin/snapshot_archive<br />
<br />
<pre>scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup <br />
vi /usr/local/etc/rsync.backup <br />
backup1 > backup3</pre><br />
<br />
crontab -e<br />
1 0 * * * /usr/local/etc/rsync.backup<br />
<br />
* edit sshd_config for security<br />
<pre>vi /etc/ssh/sshd_config<br />
ListenAddress 66.181.18.1<br />
ListenAddress 69.55.229.2<br />
ListenAddress 10.1.2.1<br />
<br />
kill -1 `cat /var/run/sshd.pid`</pre><br />
<br />
* raid chk<br />
<br />
<pre>cat > /usr/local/sbin/lsiraidchk<br />
#!/usr/bin/perl<br />
<br />
my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;<br />
<br />
foreach (@out) {<br />
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }<br />
#print $_;<br />
}</pre><br />
<br />
* netflow stuff<br />
add crontab entries<br />
<pre>crontab -e<br />
30 3 * * * /usr/local/etc/rsync.backup<br />
0 0 1 * * /sbin/ipfw zero<br />
0 0 1 * * /sbin/ipfw del 3 4 5<br />
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count<br />
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count<br />
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl; <br />
*/5 * * * * /usr/local/sbin/lsiraidchk<br />
<br />
#10 0 * * * rm /var/spool/clientmqueue/*</pre><br />
<br />
<pre>scp /etc/makefwrules.pl user@64.163.14.48:~<br />
scp /etc/makepiperules.pl user@64.163.14.48:~<br />
mv /home/user/makefwrules.pl /etc<br />
mv /home/user/makepiperules.pl /etc<br />
touch /etc/firewall.sh<br />
mkdir /etc/oldrules/</pre><br />
<br />
other binaries<br />
<br />
<pre>scp /usr/local/bin/rulemaker user@64.163.14.48:~<br />
mv ~user/rulemaker /usr/local/sbin<br />
scp ~user/Sendmail.pm user@64.163.14.48:~<br />
scp ~user/doswatch.pl user@64.163.14.48:~</pre><br />
<br />
* add nat rules<br />
<pre>vi /etc/ipnat.rules<br />
# sample entry<br />
bimap fxp0 10.1.6.70/32 -> 10.1.6.59/32<br />
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32<br />
<br />
ipnat -C -f /etc/ipnat.rules</pre><br />
<br />
* shell for user<br />
<pre>cp /root/.cshrc ~user/<br />
vi ~user/</pre><br />
change # to $<br />
<br />
* mrtg<br />
<br />
<pre>cd /usr/ports/net-mgmt/mrtg<br />
make install clean</pre><br />
(no FONTCONFIG, v3)<br />
<br />
this didn't work cause of libtool incompat<br />
<br />
so manually moved files:<br />
<br />
<pre>scp /usr/local/bin/cfgmaker user@nat2:/usr/local/bin/cfgmaker<br />
scp /usr/local/lib/perl5/site_perl/5.6.1/MRTG_lib.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/<br />
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_util.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/<br />
scp /usr/local/lib/perl5/site_perl/5.6.1/BER.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/<br />
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_Session.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/<br />
scp /usr/local/bin/mrtg root@nat2:/usr/local/bin/mrtg<br />
scp /usr/local/lib/perl5/site_perl/5.6.1/locales_mrtg.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/locales_mrtg.pm<br />
scp /usr/local/bin/rrdtool root@nat2:/usr/local/bin/rrdtool<br />
scp /usr/local/lib/perl5/site_perl/5.6.1/mach/RRDs.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/RRDs.pm<br />
rsync -av /usr/local/lib/perl5/site_perl/5.6.1/mach/auto/RRDs/ root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/auto/RRDs/<br />
scp /usr/lib/libz.so.2 root@nat2:/usr/lib/libz.so.2<br />
scp /usr/lib/libm.so.2 root@nat2:/usr/lib/libm.so.2<br />
rsync -av /usr/local/lib/librrd* root@nat2:/usr/local/lib/<br />
scp /usr/lib/libc.so.4 root@nat2:/usr/lib/libc.so.4<br />
<br />
rsync -av /usr/ports/net/rrdtool root@nat2:/usr/ports/net<br />
cd /usr/ports/net/rrdtool<br />
make install<br />
<br />
mkdir -p /mnt/data1/mrtg/data<br />
scp /usr/local/www/mgmt/mrtg/template.pl root@nat2:/mnt/data1/mrtg/<br />
scp /usr/local/www/mgmt/mrtg/host.pl root@nat2:/mnt/data1/mrtg/<br />
<br />
cfgmaker --if-template=template.pl --show-op-down --global "options[_]: growright,bits" --global 'WorkDir: /mnt/data1/mrtg/data' --global 'Interval: 1' --global 'LogFormat: rrdtool' --global 'PathAdd: /usr/local/bin' --global 'LibAdd: /usr/local/lib' --host-template=host.pl jc292401@10.1.2.50 --output=switch-p20.cfg<br />
<br />
cat > /mnt/data1/mrtg/mrtg.sh<br />
#!/bin/sh<br />
/usr/local/bin/mrtg /mnt/data1/mrtg/switch-p20.cfg<br />
<br />
chmod 0700 /mnt/data1/mrtg/mrtg.sh<br />
<br />
crontab -e<br />
* * * * * /mnt/data1/mrtg/mrtg.sh 2>&1 > /dev/null</pre><br />
<br />
* snmp firewall block<br />
<pre>cat > /usr/local/etc/rc.d/boot.sh<br />
ipfw add 10 allow udp from 69.55.230.2 to any 161<br />
ipfw add 10 allow udp from 10.1.2.1 to any 161<br />
ipfw add 11 deny udp from any to any 161<br />
chmod 0700 /usr/local/etc/rc.d/boot.sh</pre><br />
<br />
= bwdb =<br />
== Summary ==<br />
This machine tracks and stores network traffic (netflow) at castle. It is our means to monitor customer bandwidth usage.<br />
<br />
* Location: castle, cab 3-7<br />
* OS: FreeBSD 4.10 x86<br />
* Networking: Priv IP: 10.1.4.203 There are 2 onboard nic's, one of which is the "listener"<br />
* Hardware: Custom 1U. Single power supply. <br />
* Drives: one 250 GB (2 x 250GB) RAID1 array running on a Promise IDE RAID card.<br />
<br />
== Services Provided ==<br />
* netflow<br />
* mysql<br />
* bigbrother <br />
* snmp<br />
<br />
== netflow ==<br />
<br />
The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.<br />
<br />
A cronjob moves that flow file (or files if there are multiple due to some delay) <br />
1,16,31,46 * * * * /usr/home/flowbin/queue.pl<br />
<br />
into a processing queue:<br />
<tt>/usr/home/working</tt><br />
<br />
Then a separate file processes whatever flow files it finds there, inserting the data into the local mysql database:<br />
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl<br />
<br />
== mysql ==<br />
<br />
The database storing all the traffic data is named <tt>traffic</tt><br />
Tables:<br />
<pre>mysql> show tables;<br />
+---------------------------+<br />
| Tables_in_traffic |<br />
+---------------------------+<br />
| dailyIpTotals_69_55_224 |<br />
| dailyIpTotals_69_55_225 |<br />
| dailyIpTotals_69_55_226 |<br />
| dailyIpTotals_69_55_227 |<br />
| dailyIpTotals_69_55_228 |<br />
| dailyIpTotals_69_55_229 |<br />
| dailyIpTotals_69_55_230 |<br />
| dailyIpTotals_69_55_231 |<br />
| dailyIpTotals_69_55_232 |<br />
| dailyIpTotals_69_55_233 |<br />
| dailyIpTotals_69_55_234 |<br />
| dailyIpTotals_69_55_235 |<br />
| dailyIpTotals_69_55_236 |<br />
| dailyIpTotals_69_55_237 |<br />
| dailyIpTotals_69_55_238 |<br />
| dailyIpTotals_69_55_239 |<br />
| dailyPortTotals_69_55_224 |<br />
| dailyPortTotals_69_55_225 |<br />
| dailyPortTotals_69_55_226 |<br />
| dailyPortTotals_69_55_227 |<br />
| dailyPortTotals_69_55_228 |<br />
| dailyPortTotals_69_55_229 |<br />
| dailyPortTotals_69_55_230 |<br />
| dailyPortTotals_69_55_231 |<br />
| dailyPortTotals_69_55_232 |<br />
| dailyPortTotals_69_55_233 |<br />
| dailyPortTotals_69_55_234 |<br />
| dailyPortTotals_69_55_235 |<br />
| dailyPortTotals_69_55_236 |<br />
| dailyPortTotals_69_55_237 |<br />
| dailyPortTotals_69_55_238 |<br />
| dailyPortTotals_69_55_239 |<br />
| ipTotals_69_55_224 |<br />
| ipTotals_69_55_225 |<br />
| ipTotals_69_55_226 |<br />
| ipTotals_69_55_227 |<br />
| ipTotals_69_55_228 |<br />
| ipTotals_69_55_229 |<br />
| ipTotals_69_55_230 |<br />
| ipTotals_69_55_231 |<br />
| ipTotals_69_55_232 |<br />
| ipTotals_69_55_233 |<br />
| ipTotals_69_55_234 |<br />
| ipTotals_69_55_235 |<br />
| ipTotals_69_55_236 |<br />
| ipTotals_69_55_237 |<br />
| ipTotals_69_55_238 |<br />
| ipTotals_69_55_239 |<br />
| portTotals_69_55_224 |<br />
| portTotals_69_55_225 |<br />
| portTotals_69_55_226 |<br />
| portTotals_69_55_227 |<br />
| portTotals_69_55_228 |<br />
| portTotals_69_55_229 |<br />
| portTotals_69_55_230 |<br />
| portTotals_69_55_231 |<br />
| portTotals_69_55_232 |<br />
| portTotals_69_55_233 |<br />
| portTotals_69_55_234 |<br />
| portTotals_69_55_235 |<br />
| portTotals_69_55_236 |<br />
| portTotals_69_55_237 |<br />
| portTotals_69_55_238 |<br />
| portTotals_69_55_239 |<br />
+---------------------------+<br />
</pre><br />
<br />
So as you see we store each class-C block in its own table, for efficiency. Further, we store and organize data in 4 ways: "daily" tables and 15-minute granularity tables, and for each of those we track simple IP traffic and port-specific traffic. The daily tables contains 2 entries (one for each direction) for each IP for each day. For the current day, the row data is incremented as the day goes on. <br />
<br />
<pre>mysql> describe dailyIpTotals_69_55_224;<br />
+-----------+-------------+------+-----+---------+-------+<br />
| Field | Type | Null | Key | Default | Extra |<br />
+-----------+-------------+------+-----+---------+-------+<br />
| id | varchar(23) | | PRI | | |<br />
| date | date | YES | | NULL | |<br />
| ip | varchar(15) | YES | MUL | NULL | |<br />
| direction | tinyint(1) | YES | | NULL | |<br />
| octets | bigint(12) | YES | | NULL | |<br />
| packets | int(11) | YES | | NULL | |<br />
+-----------+-------------+------+-----+---------+-------+<br />
<br />
mysql> select * from dailyIpTotals_69_55_224 limit 1\G<br />
*************************** 1. row ***************************<br />
id: 6955224194-20100917-1<br />
date: 2010-09-17<br />
ip: 69.55.224.194<br />
direction: 1<br />
octets: 8821<br />
packets: 91<br />
</pre><br />
<br />
The <tt>id</tt> is a unique identifier (key), <tt>direction</tt> indicates incoming or outgoing traffic (outbound = 2, inbound = 1), <tt>octets</tt> are the amount of traffic in kilobytes, and <tt>packets</tt> is the total number of packets.<br />
<br />
The 15-minute table has similar information, but it's organized in 15 minute increments:<br />
<br />
<pre>mysql> describe ipTotals_69_55_224;<br />
+-----------+------------+------+-----+---------+-------+<br />
| Field | Type | Null | Key | Default | Extra |<br />
+-----------+------------+------+-----+---------+-------+<br />
| date | datetime | YES | | NULL | |<br />
| ip | char(15) | YES | MUL | NULL | |<br />
| direction | tinyint(1) | YES | | NULL | |<br />
| octets | bigint(20) | YES | | NULL | |<br />
| packets | int(11) | YES | | NULL | |<br />
+-----------+------------+------+-----+---------+-------+<br />
<br />
mysql> select * from ipTotals_69_55_224 limit 2\G<br />
*************************** 1. row ***************************<br />
date: 2010-01-11 19:30:00<br />
ip: 69.55.224.13<br />
direction: 1<br />
octets: 288<br />
packets: 6<br />
*************************** 2. row ***************************<br />
date: 2010-01-11 19:30:00<br />
ip: 69.55.224.12<br />
direction: 1<br />
octets: 216<br />
packets: 4</pre><br />
<br />
So for a given IP, there will be 192 rows in a given day: 4 rows per hour, *2 for 2 directions, *24 for 24hours in a day. Obviously this table is large which is why we broke it down into a daily table for quick, easy, daily-summary access.<br />
<br />
That covers the simple traffic tabulation tables. We also track traffic by port:<br />
<br />
<pre>mysql> describe dailyPortTotals_69_55_224;<br />
+-----------+-------------+------+-----+---------+-------+<br />
| Field | Type | Null | Key | Default | Extra |<br />
+-----------+-------------+------+-----+---------+-------+<br />
| id | varchar(28) | | PRI | | |<br />
| date | date | YES | | NULL | |<br />
| ip | varchar(15) | YES | MUL | NULL | |<br />
| direction | tinyint(1) | YES | | NULL | |<br />
| protocol | smallint(3) | YES | | NULL | |<br />
| port | int(11) | YES | | NULL | |<br />
| octets | bigint(11) | YES | | NULL | |<br />
| packets | int(11) | YES | | NULL | |<br />
+-----------+-------------+------+-----+---------+-------+<br />
8 rows in set (0.00 sec)<br />
<br />
mysql> select * from dailyPortTotals_69_55_224 limit 1\G<br />
*************************** 1. row ***************************<br />
id: 695522496-20091218-1-6-23<br />
date: 2009-12-18<br />
ip: 69.55.224.96<br />
direction: 1<br />
protocol: 6<br />
port: 23<br />
octets: 1796<br />
packets: 30<br />
<br />
mysql> select * from portTotals_69_55_224 limit 1\G<br />
*************************** 1. row ***************************<br />
date: 2010-09-07 18:45:00<br />
ip: 69.55.224.254<br />
direction: 1<br />
protocol: 6<br />
port: 99999<br />
octets: 144<br />
packets: 3<br />
<br />
</pre><br />
<br />
This is largely the same with 2 more additions: <tt>protocol</tt> (1=ICMP, 6=TCP, 17=UDP), and <tt>port</tt> which we set to 99999 if the traffic is return traffic and the port is above 1024. Obviously the potential for number of rows grows quickly when you consider the addition of port and protocol tracking per IP.<br />
<br />
== Regular maintenance ==<br />
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]<br />
* archive data from database<br />
archive_daily.pl 2012 09<br />
This will archive data for the given year and month from the daily summary tables. Generally we want to have a year of history in the database.<br />
<br />
archive_15min.pl 2012 09<br />
This will archive data for the given year and month from the 15min-increment tables. Generally, we want to have 6 months of history in the database.<br />
<br />
* if space becomes tight, move flow files and exported data to a backup server, both located in <tt>/usr/home/flowbin/archive</tt> and <tt>/usr/home/exported</tt>, respectively<br />
<br />
== Slaving ==<br />
<br />
If we were going to setup traffic database slaving (we don't do this anymore), perhaps cause the bwdb machine gets busy and it cannot handle traffic requests and netflow, here's how it's done:<br />
<br />
On the traffic master:<br />
<br />
GRANT REPLICATION SLAVE ON *.* TO 'repl'@'10.1.4.8' IDENTIFIED BY 'qERUG8wf';<br />
<br />
in my.cnf:<br />
<pre>bin-log<br />
server-id=1<br />
max_binlog_size=500M<br />
expire_logs_days = 3</pre><br />
<br />
on slave:<br />
in my.cnf:<br />
<pre>server-id = 2<br />
master-host = 10.1.4.203<br />
master-user = repl<br />
master-password = qERUG8wf<br />
master-connect-retry=60<br />
replicate-wild-do-table=traffic.daily%<br />
max_relay_log_size=500M<br />
expire_logs_days = 3<br />
<br />
replicate-wild-do-table=traffic.%</pre><br />
<br />
on master:<br />
touch /usr/home/working/.lock<br />
(make sure processflows not running)<br />
<br />
<pre>FLUSH TABLES WITH READ LOCK;<br />
cd /usr/home/database/traffic<br />
tar -czf mysql-traffic-snapshot.tgz ./daily*<br />
(~1G)<br />
SHOW MASTER STATUS;<br />
+-----------------+-----------+--------------+------------------+<br />
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |<br />
+-----------------+-----------+--------------+------------------+<br />
| bwdb-bin.000039 | 154432615 | | |<br />
+-----------------+-----------+--------------+------------------+<br />
<br />
(write down info)<br />
UNLOCK TABLES;<br />
scp mysql-traffic-snapshot.tgz 10.1.4.5:/mnt/data1/db/mysql/traffic/</pre><br />
<br />
on slave:<br />
<pre>mkdir /mnt/data1/db/mysql/traffic<br />
cd /mnt/data1/db/mysql/traffic/<br />
tar xzvf mysql-traffic-snapshot.tgz<br />
(restart mysql)<br />
CHANGE MASTER TO MASTER_HOST='10.1.4.203',MASTER_USER='repl',MASTER_PASSWORD='qERUG8wf',MASTER_LOG_FILE='bwdb-bin.000059',MASTER_LOG_POS=482502186;<br />
START SLAVE;</pre><br />
<br />
<pre>cd /usr/home/database/traffic<br />
scp *</pre> <br />
<br />
<pre>optimize table dailyPortTotals_69_55_224;<br />
optimize table dailyPortTotals_69_55_225;<br />
optimize table dailyPortTotals_69_55_226;<br />
optimize table dailyPortTotals_69_55_227;<br />
optimize table dailyPortTotals_69_55_228;<br />
optimize table dailyPortTotals_69_55_229;<br />
optimize table dailyPortTotals_69_55_230;<br />
optimize table dailyPortTotals_69_55_231;<br />
optimize table dailyPortTotals_69_55_232;<br />
optimize table dailyPortTotals_69_55_233;<br />
optimize table dailyPortTotals_69_55_234;<br />
optimize table dailyPortTotals_69_55_235;<br />
optimize table dailyPortTotals_69_55_236;<br />
optimize table dailyPortTotals_69_55_237;<br />
optimize table dailyPortTotals_69_55_238;<br />
optimize table dailyPortTotals_69_55_239;</pre><br />
<br />
== Build ==<br />
<br />
=== BIOS Config ===<br />
disable quiet boot<br />
<br />
set to last state after power loss<br />
<br />
set date/time to GMT<br />
<br />
enable serial console output (baud rate 115200)<br />
<br />
=== Install OS ===<br />
<br />
Install FreeBSD 8.3 amd64<br />
<br />
* partition map:<br />
<pre>/ 500m<br />
swap 4096m<br />
/var 256m<br />
/tmp 256m<br />
/usr ~</pre><br />
<br />
* edit /etc/make.conf <br />
Castle:<br />
<pre>echo "WITHOUT_X11=yes \<br />
KERNCONF=bwdb \<br />
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre><br />
<br />
i2b:<br />
<pre>echo "WITHOUT_X11=yes \<br />
KERNCONF=bwdb2 \<br />
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre><br />
<br />
* add settings to /boot/loader.conf and /boot.config<br />
<br />
<pre>echo "-Dh" >> /boot.config<br />
<br />
echo 'console="comconsole,vidconsole" \<br />
boot_multicons="YES" \<br />
boot_serial="YES" \<br />
comconsole_speed="115200"' >> /boot/loader.conf</pre><br />
<br />
* turn off all ttyv's except 0 and 1 in /etc/ttys<br />
also turn on ttyu0, change type to vt100:<br />
<pre>vi /etc/ttys<br />
<br />
ttyv2 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv3 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv4 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv5 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv6 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv7 "/usr/libexec/getty Pc" cons25 off secure<br />
# Serial terminals<br />
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.<br />
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure<br />
<br />
kill -1 1</pre><br />
<br />
on console server:<br />
vi /etc/remote<br />
(rename port to jail8 depending on where and which digi plugged into)<br />
test serial console<br />
<br />
<br />
* populate hosts<br />
i2b:<br />
<pre>echo "69.55.230.10 backup2" >> /etc/hosts<br />
echo "69.55.230.11 backup1" >> /etc/hosts<br />
echo "10.1.2.3 backup3" >> /etc/hosts</pre><br />
<br />
castle:<br />
<pre>echo "10.1.4.3 backup2 backup2.johncompanies.com" >> /etc/hosts<br />
echo "10.1.4.8 backup1 backup1.johncompanies.com" >> /etc/hosts<br />
echo "10.1.4.4 mail mail.johncompanies.com" >> /etc/hosts<br />
</pre><br />
<br />
* put key in authorized_keys on backup1 and backup2<br />
cd<br />
ssh-keygen -t dsa -b 1024 <br />
(default location, leave password blank)<br />
<br />
castle:<br />
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys' <br />
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' <br />
<br />
i2b:<br />
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys' <br />
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' <br />
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys' <br />
<br />
confirm that you can ssh to backup2 and backup1 (and backup3 if at i2b) without getting a login prompt<br />
<br />
ssh backup1 hostname<br />
ssh backup2 hostname<br />
<br />
* edit root's path and login script:<br />
vi /root/.cshrc<br />
<br />
Change alias entries (add G):<br />
<pre>alias la ls -aG<br />
alias lf ls -FAG<br />
alias ll ls -lAG<br />
alias ls ls -AG<br />
</pre><br />
<br />
and alter the prompt, set the following:<br />
set prompt = "`/bin/hostname -s` %/# "<br />
<br />
* install cvsup<br />
cd /usr/ports/net/cvsup-without-gui <br />
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null<br />
<br />
* get latest sources for this release:<br />
<pre>cd /usr/src <br />
echo "*default host=cvsup4.freebsd.org\<br />
*default base=/usr\<br />
*default prefix=/usr\<br />
*default release=cvs tag=RELENG_8_3\<br />
*default delete use-rel-suffix\<br />
*default compress\<br />
src-all" > sup<br />
<br />
cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null</pre><br />
<br />
* configure new kernel<br />
<br />
cd /usr/src/sys/amd64/conf <br />
scp backup2:/mnt/data4/build/freebsd/kern_config-bwdb-8.3-amd64 ./bwdb<br />
<br />
Edit config and change name:<br />
vi bwdb<br />
ident bwdb<br />
<br />
* build, install kernel and world<br />
<br />
<pre>cd /boot<br />
<br />
mv kernel kernel.GENERIC<br />
cd kernel.GENERIC<br />
cd /usr/src<br />
make buildkernel installkernel<br />
<br />
make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null<br />
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)<br />
make installworld <br />
(2450: 3min, supermicro: 1min, 2950: :34)<br />
mergemaster -i</pre><br />
<br />
* populate /etc/rc.conf with IPs and NFS settings<br />
<br />
castle:<br />
<pre>vi /etc/rc.conf<br />
<br />
hostname="bwdb.johncompanies.com"<br />
kern_securelevel_enable="NO"<br />
portmap_enable="NO"<br />
sendmail_enable="NO"<br />
usbd_enable="YES"<br />
<br />
xntpd_enable="YES"<br />
nfs_client_enable="YES"<br />
nfs_reserved_port_only="YES"<br />
ifconfig_fxp0="inet 10.1.4.203 netmask 255.255.255.0"<br />
ifconfig_em0="up promisc"<br />
defaultrouter="10.1.4.1"<br />
snmpd_enable="YES"<br />
<br />
inetd_enable="YES"<br />
inetd_flags="-wW -a 10.1.4.203"<br />
fsck_y_enable="YES"<br />
background_fsck="NO"<br />
sshd_enable="YES"<br />
ipfw_load="YES"</pre><br />
<br />
i2b:<br />
<pre>vi /etc/rc.conf<br />
<br />
hostname="bwdb2.johncompanies.com"<br />
kern_securelevel_enable="NO"<br />
portmap_enable="NO"<br />
sendmail_enable="NO"<br />
usbd_enable="YES"<br />
<br />
xntpd_enable="YES"<br />
nfs_client_enable="YES"<br />
nfs_reserved_port_only="YES"<br />
ifconfig_fxp0="inet 10.1.2.4 netmask 255.255.255.0"<br />
ifconfig_em0="up promisc"<br />
defaultrouter="10.1.2.1"<br />
snmpd_enable="YES"<br />
<br />
inetd_enable="YES"<br />
inetd_flags="-wW -a 10.1.2.4"<br />
fsck_y_enable="YES"<br />
background_fsck="NO"<br />
sshd_enable="YES"<br />
ipfw_load="YES"</pre><br />
<br />
* reboot. Confirm new kernel is loaded<br />
<br />
uname -a<br />
<br />
* update ports:<br />
<pre>cd /usr/ports<br />
echo "*default host=cvsup4.FreeBSD.org\<br />
*default base=/usr\<br />
*default prefix=/usr\<br />
*default release=cvs tag=RELENG_8_3\<br />
*default delete use-rel-suffix\<br />
*default compress\<br />
ports-all tag=." > sup<br />
<br />
cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre><br />
<br />
* Install raid mgmt tool<br />
<br />
<pre>cd /usr/local/sbin<br />
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz .<br />
tar xzf tw_cli-freebsd-x86_64-9.5.0.1.tgz<br />
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz<br />
chmod 0700 tw_cli</pre><br />
<br />
Test:<br />
./tw_cli info c0<br />
<br />
Grab raid check script:<br />
scp backup1:/usr/local/sbin/3wraidchk /usr/local/etc<br />
<br />
Setup cronjob:<br />
<pre>crontab -e<br />
*/5 * * * * /usr/local/etc/3wraidchk</pre><br />
<br />
* install rsync from ports<br />
cd /usr/ports/net/rsync<br />
make install clean<br />
<br />
choose default options<br />
<br />
* install perl from ports<br />
cd /usr/ports/lang/perl5.8<br />
make install clean<br />
<br />
choose default options<br />
<br />
* install bb client<br />
<br />
Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux: <br />
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/<br />
<br />
adduser<br />
<br />
Output/response:<br />
<br />
<pre>Username: bb<br />
Full name: bb<br />
Uid (Leave empty for default): 1984<br />
Login group [bb]:<br />
Login group is bb. Invite bb into other groups? []:<br />
Login class [default]:<br />
Shell (sh csh tcsh nologin) [sh]: <br />
Home directory [/home/bb]:<br />
Use password-based authentication? [yes]:<br />
Use an empty password? (yes/no) [no]:<br />
Use a random password? (yes/no) [no]: yes<br />
Lock out the account after creation? [no]:<br />
Username : bb<br />
Password : <random><br />
Full Name : bb<br />
Uid : 1984<br />
Class :<br />
Groups : bb<br />
Home : /home/bb<br />
Shell : /bin/sh<br />
Locked : no<br />
OK? (yes/no): yes</pre><br />
<br />
cd /usr/home/bb<br />
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .<br />
tar xzf bb-freebsd_linuxcompat.tgz<br />
<br />
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:<br />
<br />
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \<br />
10.1.4.203 bwdb.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts<br />
<br />
Edit for machine name and private IP.<br />
<br />
if this machine is at i2b:<br />
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \<br />
10.1.2.4 bwdb2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts<br />
<br />
<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles <br />
<br />
MACHINE="bwdb,johncompanies,com" # HAS TO BE IN A,B,C FORM</pre><br />
Edit for machine name.<br />
<br />
Have bb watch for flow-capture, mysql<br />
<pre>cat >> /home/bb/bbc1.9e-btf/etc/bb-proctab<br />
localhost: flow-capture :<br />
localhost: mysqld :</pre><br />
<br />
<br />
<pre>cd /usr/home/bb/bbc1.9e-btf/etc<br />
./bbchkcfg.sh <br />
(y to questions)<br />
./bbchkhosts.sh<br />
(ignore ssh errors)<br />
cd ../..<br />
chown -R bb .<br />
su bb<br />
cd<br />
cd bbc1.9e-btf<br />
./runbb.sh start<br />
more BBOUT <br />
(look for errors)<br />
exit</pre><br />
<br />
Put in script to start bb @ boot:<br />
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh<br />
chmod +x /usr/local/etc/rc.d/bb.sh<br />
<br />
<br />
If this is at i2b, punch a hole in the firewall to allow it to communicate with bb monitor:<br />
<br />
ipfw add 00096 allow tcp from 66.181.18.0/27 to 69.55.230.2<br />
<br />
<br />
* configure bb on mail<br />
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts<br />
10.1.4.203 bwdb.johncompanies.com # ssh<br />
<br />
su bb<br />
cd<br />
bbsrc/bb/runbb.sh restart ; exit</pre><br />
<br />
* configure ntp server<br />
Castle:<br />
echo "server 10.1.4.1" > /etc/ntp.conf<br />
<br />
I2b:<br />
echo "server 10.1.2.1" > /etc/ntp.conf<br />
<br />
<pre>/usr/sbin/ntpd -p /var/run/ntpd.pid<br />
sleep 2; ntpq -p</pre><br />
(confirm it’s able to reach our time server)<br />
<br />
<pre>echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh<br />
chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre><br />
<br />
* fwd and reverse lookups on ns1c<br />
vr johncompanies.com<br />
(edit the PTR too)<br />
<br />
* setup backups<br />
<pre>echo '#\!/bin/sh\<br />
backupdir=/data/bwdb/current\<br />
server=backup1\<br />
\<br />
## ENTRY /etc\<br />
## ENTRY /usr/home/flowbin\<br />
## ENTRY /usr/home/database' > /usr/local/etc/backup.config</pre><br />
<br />
Castle:<br />
setup backup dirs:<br />
ssh backup1 mkdir -p /data/bwdb/current<br />
on backup1, add the system to <br />
vi /usr/local/sbin/snapshot_rotate<br />
<br />
I2b:<br />
setup backup dirs:<br />
ssh backup3 mkdir -p /data/bwdb/current<br />
on backup3, add the system to <br />
vi /usr/local/sbin/snapshot_archive<br />
<br />
<br />
Copy over the backup script:<br />
scp backup2:/d4/bin/freebsd8.x/rsync.backup /usr/local/etc/<br />
<br />
Edit rsync.backup and change <tt>config</tt> var to point to correct config file location: <tt>/usr/local/etc/backup.config</tt><br />
<br />
<pre>crontab -e<br />
5 0 * * * /usr/local/etc/rsync.backup</pre><br />
<br />
* make /root/logs<br />
mkdir /root/logs<br />
<br />
* edit sshd_config for security<br />
<br />
<pre>vi /etc/ssh/sshd_config<br />
ListenAddress 10.1.4.203<br />
PermitRootLogin yes<br />
<br />
kill -1 `cat /var/run/sshd.pid`</pre><br />
<br />
Edit for private IP.<br />
<br />
* snmp<br />
<br />
(Before doing this you may need to take down the firewall and also add to resolv.conf 69.43.143.41)<br />
<pre><br />
cd /usr/ports/net-mgmt/net-snmp<br />
make install clean<br />
(defaults)<br />
<br />
cat >> /etc/rc.conf<br />
snmpd_enable="YES"<br />
snmpd_flags="-a"<br />
snmpd_conffile="/usr/local/share/snmp/snmpd.conf"<br />
snmptrapd_enable="YES"<br />
snmptrapd_flags="-a -p /var/run/snmptrapd.pid"<br />
<br />
cat > /usr/local/share/snmp/snmpd.conf<br />
rocommunity jcread 10.1.4.5<br />
rocommunity jcread 10.1.4.202<br />
</pre><br />
<br />
=== netflow ===<br />
<br />
Install flow tools:<br />
<pre>cd /usr/ports/net-mgmt/flow-tools<br />
make install clean</pre><br />
Defaults.<br />
<br />
mkdir /usr/home/flows<br />
<br />
Flow start script:<br />
echo "/usr/local/bin/flow-capture -w /usr/home/flows -S5 -N -2 0/10.1.4.203/4444" > /usr/local/etc/rc.d/flow-capture.sh<br />
<br />
chmod 0700 /usr/local/etc/rc.d/flow-capture.sh<br />
<br />
Edit for private IP.<br />
<br />
Netgraph start script:<br />
<pre><br />
cat > /usr/local/etc/rc.d/netgraph.sh<br />
<br />
/usr/sbin/ngctl -f- <<-SEQ<br />
mkpeer em0: netflow lower iface0<br />
name em0:lower netflow<br />
connect em0: netflow: upper out0<br />
mkpeer netflow: ksocket export inet/dgram/udp<br />
msg netflow:export connect inet/10.1.4.203:4444<br />
SEQ<br />
<br />
#/usr/sbin/ngctl -f- <<-SEQ<br />
#shutdown netflow:<br />
#SEQ<br />
<br />
chmod 0700 /usr/local/etc/rc.d/netgraph.sh</pre><br />
Edit for private IP.<br />
<br />
Confirm netflow is running after running scripts:<br />
<pre>newbwdb /usr/ports/net-mgmt/flow-tools# /usr/sbin/ngctl<br />
Available commands:<br />
config get or set configuration of node at <path><br />
connect Connects hook <peerhook> of the node at <relpath> to <hook><br />
debug Get/set debugging verbosity level<br />
dot Produce a GraphViz (.dot) of the entire netgraph.<br />
help Show command summary or get more help on a specific command<br />
list Show information about all nodes<br />
mkpeer Create and connect a new node to the node at "path"<br />
msg Send a netgraph control message to the node at "path"<br />
name Assign name <name> to the node at <path><br />
read Read and execute commands from a file<br />
rmhook Disconnect hook "hook" of the node at "path"<br />
show Show information about the node at <path><br />
shutdown Shutdown the node at <path><br />
status Get human readable status information from the node at <path><br />
types Show information about all installed node types<br />
write Send a data packet down the hook named by "hook".<br />
quit Exit program<br />
+ show netflow:<br />
Name: netflow Type: netflow ID: 00000004 Num hooks: 3<br />
Local hook Peer name Peer type Peer ID Peer hook<br />
---------- --------- --------- ------- ---------<br />
export <unnamed> ksocket 00000005 inet/dgram/udp<br />
out0 em0 ether 00000001 upper<br />
iface0 em0 ether 00000001 lower<br />
+<br />
</pre><br />
<br />
We notice that sometimes flow-capture is failing due to swap exhaustion (even after adding more swap). So we crontab flow-capture to restart (it's ok to start if it's already running, it just quits):<br />
<br />
<pre><br />
crontab -e<br />
#restart flow-capture<br />
*/15 * * * * /usr/local/etc/rc.d/flow-capture.sh<br />
</pre><br />
<br />
==== process flow tools ==== <br />
<br />
<pre>mkdir /usr/home/flowbin<br />
mkdir /usr/home/working</pre><br />
<br />
Install modules:<br />
<pre>cd /usr/ports/devel/p5-Date-Calc<br />
make install clean<br />
cd /usr/ports/mail/p5-Mail-Sendmail<br />
make install clean</pre><br />
<br />
Queue script:<br />
<pre><br />
cat > /usr/home/flowbin/queue.pl<br />
#!/usr/bin/perl<br />
<br />
use strict;<br />
<br />
BEGIN {<br />
push @INC, "/usr/home/flowbin";<br />
}<br />
<br />
use date;<br />
<br />
my $flowbase = "/usr/home/flows";<br />
#my $flowqueue = "/usr/home/queue";<br />
my $flowqueue = "/usr/home/working";<br />
<br />
my ($date, $time) = date::CurrentDateTime();<br />
<br />
my $flowdir = mkFlowDir($date);<br />
`mv $flowdir/ft-* $flowqueue`;<br />
<br />
if (date::DateWindow($date, $time, $date, "00:00:00", 600)) {<br />
my $newdate = date::AddDays($date, -1);<br />
my $flowdir = mkFlowDir($newdate);<br />
`mv $flowdir/ft-* $flowqueue`;<br />
}<br />
<br />
sub mkFlowDir {<br />
my $date = shift;<br />
$date =~ /([0-9]{4}-[0-9]{2})/;<br />
my $yearmonth = $1;<br />
return "$flowbase/$yearmonth/$date";<br />
}<br />
</pre><br />
<br />
Date.pm module:<br />
<pre><br />
cat > /usr/home/flowbin/date.pm<br />
#!/usr/local/bin/perl<br />
#<br />
# $Header: /usr/cvs/newgw/lib/date.pm,v 1.2 2003/11/24 17:06:02 glenn Exp $<br />
#<br />
# Copyright (c) 2001, 2002, 2003<br />
# e-Monitoring Networks, Inc. All rights reserved.<br />
#<br />
#<br />
#<br />
# date.pl - Higher level functions written on top of Date::Calc<br />
<br />
package date;<br />
<br />
use strict;<br />
use Date::Calc qw(:all);<br />
<br />
sub DayDiff { #calculate the difference in days from two dates<br />
my $date1 = shift;<br />
my $date2 = shift;<br />
my ($year1, $month1, $day1) = &DateToymd($date1);<br />
my ($year2, $month2, $day2) = &DateToymd($date2);<br />
my $diff = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);<br />
return $diff;<br />
}<br />
<br />
sub AddDays { #adds specified number of days to the supplied date<br />
my $date = shift;<br />
my $days = shift;<br />
my ($year, $month, $day) = &DateToymd($date);<br />
my ($nyear, $nmonth, $nday) = &Add_Delta_Days($year, $month, $day, $days);<br />
my $ndate = &ymdToDate($nyear, $nmonth, $nday);<br />
return $ndate;<br />
}<br />
<br />
sub AddHours { #adds specified number of hours to the supplied date and time<br />
my $date = shift;<br />
my $time = shift;<br />
my $addhours = shift;<br />
my $adddays = 0;<br />
if (abs($addhours / 24) >= 1) {<br />
$adddays = int($addhours / 24);<br />
$addhours -= $adddays * 24;<br />
}<br />
my ($year, $month, $day) = &DateToymd($date);<br />
my ($hour, $minute, $second) = &TimeTohms($time);<br />
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,<br />
$hour, $minute, $second,<br />
$adddays, $addhours, 0, 0);<br />
my $ndate = &ymdToDate($ny, $nm, $nd);<br />
my $ntime = &hmsToTime($nh, $nmin, $ns);<br />
return $ndate, $ntime;<br />
}<br />
<br />
sub AddMinutes {<br />
my $date = shift;<br />
my $time = shift;<br />
my $minutes = shift;<br />
my ($year, $month, $day) = &DateToymd($date);<br />
my ($hour, $minute, $second) = &TimeTohms($time);<br />
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,<br />
$hour, $minute, $second,<br />
0, 0, $minutes, 0);<br />
my $ndate = &ymdToDate($ny, $nm, $nd);<br />
my $ntime = &hmsToTime($nh, $nmin, $ns);<br />
return $ndate, $ntime;<br />
}<br />
<br />
sub CurrentDateTime { #return the current date and time<br />
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;<br />
my $date = &ymdToDate($y, $m, $d);<br />
my $time = &hmsToTime($h, $min, $s);<br />
return $date, $time;<br />
}<br />
<br />
sub Currentymd { #return the current year, month and day as separate variables<br />
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;<br />
return $y, $m, $d;<br />
}<br />
<br />
sub DateToymd { #takes a date and returns year, month, day as individual values<br />
my $date = shift;<br />
if ($date =~ /([0-9]{4})-([0-9]{2})-([0-9]{2})/) {<br />
my $day = $3;<br />
my $month = $2;<br />
my $year = $1;<br />
return $year, $month, $day;<br />
}<br />
return undef;<br />
}<br />
<br />
sub TimeTohms { #takes a time and return hours minutes and seconds as individual values<br />
my $time = shift;<br />
if ($time =~ /([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})/) {<br />
my $hour = $1;<br />
my $minute = $2;<br />
my $second = $3;<br />
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }<br />
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }<br />
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }<br />
return $hour, $minute, $second;<br />
}<br />
return undef;<br />
}<br />
<br />
sub ymdToDate { #takes year, month, day and assembles them into our date format<br />
my $year = shift;<br />
my $month = shift;<br />
my $day = shift;<br />
if (defined($year) && defined($month) && defined ($day)) {<br />
$month = sprintf("%02d", $month);<br />
$day = sprintf("%02d", $day);<br />
return "$year-$month-$day";<br />
}<br />
return undef;<br />
}<br />
<br />
sub hmsToTime { #takes hour minute and second and assembles them into our time format<br />
my $hour = shift;<br />
my $minute = shift;<br />
my $second = shift;<br />
if (defined($hour) && defined($minute) && defined ($second)) {<br />
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }<br />
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }<br />
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }<br />
return sprintf ("%02d:%02d:%02d", $hour, $minute, $second);<br />
}<br />
return undef;<br />
}<br />
<br />
sub CompareDates { #compares two date and time pairs<br />
my $date1 = shift;<br />
my $time1 = shift;<br />
my $date2 = shift;<br />
my $time2 = shift;<br />
<br />
my ($year1, $month1, $day1) = &DateToymd($date1);<br />
my ($hour1, $minute1, $second1) = &TimeTohms($time1);<br />
my ($year2, $month2, $day2) = &DateToymd($date2);<br />
my ($hour2, $minute2, $second2) = &TimeTohms($time2);<br />
<br />
# &debug("$year1, $month1, $day1, $year2, $month2, $day2");<br />
my $days = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);<br />
if ($days > 0) { return 1;}<br />
if ($days < 0) { return -1;}<br />
if ($days == 0) { #same day, compare times<br />
my $seconds1 = $second1 + (60 * $minute1) + (3600 * $hour1);<br />
my $seconds2 = $second2 + (60 * $minute2) + (3600 * $hour2);<br />
if ($seconds1 < $seconds2) { return 1;}<br />
if ($seconds1 > $seconds2) { return -1;}<br />
if ($seconds1 == $seconds2) { return 0;}<br />
}<br />
return undef;<br />
}<br />
<br />
sub DateWindow { #compares two date time pairs to see if they are < X seconds apart<br />
my $date1 = shift;<br />
my $time1 = shift;<br />
my $date2 = shift;<br />
my $time2 = shift;<br />
my $window = shift;<br />
<br />
my ($year1, $month1, $day1) = &DateToymd($date1);<br />
my ($hour1, $minute1, $second1) = &TimeTohms($time1);<br />
my ($year2, $month2, $day2) = &DateToymd($date2);<br />
my ($hour2, $minute2, $second2) = &TimeTohms($time2);<br />
<br />
my ($day, $hour, $minute, $second) =<br />
&Delta_DHMS($year1, $month1, $day1, $hour1, $minute1, $second1,<br />
$year2, $month2, $day2, $hour2, $minute2, $second2);<br />
$minute *= 60;<br />
$hour *= 3600;<br />
$day *= 86400;<br />
my $total = $second + $minute + $hour + $day;<br />
if (abs($total) < $window) {<br />
return 1;<br />
}<br />
return 0;<br />
}<br />
<br />
sub CheckDateOrder { #takes three dates/times, returns true if they are in chronological order<br />
my $date1 = shift;<br />
my $time1 = shift;<br />
my $date2 = shift;<br />
my $time2 = shift;<br />
my $date3 = shift;<br />
my $time3 = shift;<br />
if (&CompareDates($date1, $time1, $date2, $time2) == -1) {<br />
return 0;<br />
}<br />
if (&CompareDates($date2, $time2, $date3, $time3) == -1) {<br />
return 0;<br />
}<br />
return 1;<br />
}<br />
<br />
sub EpochSeconds { #calculates number of seconds since the epoch for the given date/time<br />
my $date = shift;<br />
my $time = shift;<br />
my ($year, $month, $day) = &DateToymd($date);<br />
my ($hour, $minute, $second) = &TimeTohms($time);<br />
my ($d, $h, $m, $s) = &Delta_DHMS(1970, 1, 1, 0, 0, 0,<br />
$year, $month, $day, $hour, $minute, $second);<br />
my $seconds = $s + (60 * $m) + (3600 * $h) + (86400 * $d);<br />
return $seconds;<br />
}<br />
<br />
sub SecondsToDateTime { #converts seconds since epoch to date/time<br />
my $seconds = shift;<br />
my $days = int($seconds / 86400);<br />
$seconds -= $days * 86400;<br />
my $hours = int($seconds / 3600);<br />
$seconds -= $hours * 3600;<br />
my $minutes = int($seconds / 60);<br />
$seconds -= $minutes * 60;<br />
my ($year, $month, $day, $hour, $minute, $second) =<br />
&Add_Delta_DHMS(1970, 1, 1, 0, 0, 0, $days, $hours, $minutes, $seconds);<br />
$month = sprintf("%02d", $month);<br />
$day = sprintf("%02d", $day);<br />
$hour = sprintf("%02d", $hour);<br />
$minute = sprintf("%02d", $minute);<br />
$second = sprintf("%02d", $second);<br />
return "$year-$month-$day", "$hour:$minute:$second";<br />
}<br />
<br />
sub DateToDayName {<br />
my $date = shift;<br />
my ($year, $month, $day) = &DateToymd($date);<br />
my $name = &Day_of_Week_to_Text(&Day_of_Week($year, $month, $day));<br />
$name =~ /^[A-Za-z]{3}/;<br />
$name = $&;<br />
return $name;<br />
}<br />
<br />
sub ValiDate {<br />
return @_;<br />
<br />
<br />
<br />
}<br />
<br />
sub CheckBusinessDay { # checks to see if date is business day. 1=yes, 0=no<br />
my $date = shift;<br />
my ($year, $month, $day) = &DateToymd($date);<br />
if (Day_of_Week($year,$month,$day) < 6) { return 1; }<br />
else { return 0; }<br />
}<br />
<br />
1; #don't remove this line<br />
</pre><br />
<br />
chmod 0700 /usr/home/flowbin/queue.pl<br />
<br />
Setup cronjob:<br />
<pre>crontab -e<br />
#move flow data into the queue<br />
1,16,31,46 * * * * /usr/home/flowbin/queue.pl</pre><br />
<br />
==== flow processing: i2b ==== <br />
<pre>cat > /usr/home/flowbin/processflows-sql.pl<br />
#!/usr/bin/perl<br />
<br />
#use strict;<br />
#$debug=1;<br />
#$dry=1;<br />
<br />
my $log = '/usr/home/flowbin/discards.log';<br />
<br />
use Data::Dumper;<br />
<br />
BEGIN {<br />
push @INC, "/usr/home/flowbin";<br />
}<br />
<br />
#my $queuedir = "/usr/home/queue";<br />
my $queuedir = "/usr/home/working";<br />
my $archivedir = "/usr/home/archive";<br />
my $sqldir = "/usr/home/sql";<br />
my $sqldirworking = "/usr/home/sql/tmp";<br />
<br />
unless ($dry) {<br />
if (-e "$queuedir/.lock") {<br />
open(FILE, "$queuedir/.lock");<br />
my $pid = <FILE>;<br />
chomp($pid);<br />
close(FILE);<br />
if (kill(0, $pid)) {<br />
#another process is using the queue, bail out<br />
exit(0);<br />
}<br />
else {<br />
#dead lock file, remove it<br />
`rm $queuedir/.lock`;<br />
}<br />
}<br />
open(FILE, "> $queuedir/.lock");<br />
print FILE "$$\n";<br />
close(FILE);<br />
}<br />
<br />
opendir(DIR, $queuedir);<br />
my @files = readdir(DIR);<br />
closedir(DIR);<br />
<br />
foreach my $file (sort @files) {<br />
unless($file =~ /^\./) {<br />
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;<br />
my $date = "$1 $2:$3:$4";<br />
my $outfile = "$1-$2:$3.sql";<br />
unless (open (SQL, "+> $sqldirworking/$outfile")) { die "cant open $sqldirworking/$outfile"; }<br />
my $condensedDate = $1;<br />
$condensedDate =~ s/-//g;<br />
my $iptotal = {};<br />
my $protototal = {};<br />
my $porttotal = {};<br />
<br />
&debug("started file $file at ");<br />
&debug(`date`);<br />
&debug("getting raw flow data (flow-print)");<br />
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;<br />
&debug("aggregating data at ");<br />
&debug(`date`);<br />
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }<br />
LOOP: while (my $line = readline DATA) {<br />
my @d = split /[\s]+/, $line;<br />
if ($d[0] ne '' && $d[0] ne 'Start') {<br />
my $addr = 0;<br />
my $port = 0;<br />
<br />
#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets<br />
#0 1 2 3 4 5 6 7 8 9 10 11<br />
#|<br />
# outbound = 2, inbound = 1<br />
<br />
my (@src_ip) = split '\.', $d[3];<br />
my (@dst_ip) = split '\.', $d[6];<br />
<br />
if ($src_ip[0] == 69 && $src_ip[1] == 55 && ($src_ip[2] == 229 || $src_ip[2] == 231)) { # for i2b<br />
$d[2] = 2;<br />
# hack for outbound bulk traffic counted 2x<br />
#if ($src_ip[2] == 231) { $d[11] /= 2; $d[10] /= 2; }<br />
}<br />
# note- this is where we filter out IPs only found at i2b<br />
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && ($dst_ip[2] == 229 || $dst_ip[2] == 231)) { # for i2b<br />
$d[2] = 1;<br />
}<br />
else { next LOOP; }<br />
<br />
if ($d[2] == 2) {<br />
$addr = $d[3];<br />
# if the dst-port is low, store that<br />
if ($d[7] <= 1024) { $port = $d[7]; }<br />
# if the src-port is low, store that<br />
elsif ($d[4] <= 1024) { $port = $d[4]; }<br />
else { $port = 99999; }<br />
}<br />
elsif ($d[2] == 1) {<br />
$addr = $d[6];<br />
# if the dst-port is high, assume its return traffic, try to store src-port if low<br />
if ($d[7] > 1024) {<br />
if ($d[4] <= 1024) { $port = $d[4]; }<br />
else { $port = 99999; }<br />
} else {<br />
$port = $d[7];<br />
}<br />
} else {<br />
next LOOP;<br />
}<br />
<br />
my (@ip) = split '\.', $addr;<br />
unless ($ip[0] == 69) { next LOOP; }<br />
unless ($ip[1] == 55) { next LOOP; }<br />
unless ($ip[2] == 229 || $ip[2] == 231) { next LOOP; }<br />
<br />
my $classC = "$ip[0]_$ip[1]_$ip[2]";<br />
<br />
# IP dir<br />
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }<br />
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }<br />
#<br />
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }<br />
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }<br />
#<br />
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }<br />
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }<br />
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];<br />
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];<br />
<br />
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];<br />
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];<br />
<br />
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];<br />
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];<br />
}<br />
}<br />
close(DATA);<br />
`rm /usr/home/working/tmp-$file`;<br />
&debug("processing ip totals at ");<br />
&debug(`date`);<br />
foreach my $classC (keys(%{$iptotal})) {<br />
my @values;<br />
foreach my $ip (keys(%{$iptotal->{$classC}})) {<br />
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {<br />
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};<br />
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};<br />
# $packets = $packets > 2147483647 ? 0 : $packets;<br />
if ($octets > 2147483647) {<br />
my $ddir = $dir==1 ? 'in' : 'out';<br />
#print SQL "$date $ip $ddir $octets\n";<br />
# $octets = 0;<br />
}<br />
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction<br />
my $id = "$ip-$condensedDate-$dir";<br />
$id =~ s/\.//g;<br />
push @values, "('$date', '$ip', $dir, $octets, $packets)";<br />
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";<br />
print "$sql\n" if $dry;<br />
print SQL "$sql;\n";<br />
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");<br />
}<br />
}<br />
<br />
# break inserts into 100 records at a time<br />
&debug("inserting $#values +1 values");<br />
while ($#values > 0) {<br />
my $sql = "insert into ipTotals_$classC values ";<br />
my $max_index = $#values > 100 ? 100 : $#values;<br />
for (my $i=0; $i<=$max_index; $i++) {<br />
$sql .= shift @values;<br />
$sql .= ',';<br />
}<br />
chop $sql;<br />
print "$sql\n" if $dry;<br />
print SQL "$sql;\n";<br />
}<br />
}<br />
<br />
# &debug("processing protocol totals at ");<br />
# &debug(`date`);<br />
# foreach my $classC (keys(%{$protototal})) {<br />
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;<br />
# my @values;<br />
# foreach my $ip (keys(%{$protototal->{$classC}})) {<br />
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {<br />
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {<br />
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};<br />
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};<br />
# # $octets = $octets > 2147483647 ? 0 : $octets;<br />
# # $packets = $packets > 2147483647 ? 0 : $packets;<br />
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto<br />
# my $id = "$ip-$condensedDate-$dir-$proto";<br />
# $id =~ s/\.//g;<br />
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";<br />
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";<br />
# print "$sql\n" if $dry;<br />
# $db->query($sql) unless $dry;<br />
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");<br />
# }<br />
# }<br />
# }<br />
# $db->query("unlock tables") unless $dry;<br />
# my $sql = "insert into protoTotals_$classC values ";<br />
# $sql .= join ',', @values;<br />
# $db->query("lock tables protoTotals_$classC write") unless $dry;<br />
# print "$sql\n" if $dry;<br />
# $db->query($sql) unless $dry;<br />
# $db->query("unlock tables") unless $dry;<br />
# }<br />
<br />
&debug("processing port totals at ");<br />
&debug(`date`);<br />
foreach my $classC (keys(%{$porttotal})) {<br />
my @values;<br />
foreach my $ip (keys(%{$porttotal->{$classC}})) {<br />
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {<br />
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {<br />
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {<br />
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};<br />
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};<br />
# $octets = $octets > 2147483647 ? 0 : $octets;<br />
# $packets = $packets > 2147483647 ? 0 : $packets;<br />
<br />
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port<br />
my $id = "$ip-$condensedDate-$dir-$proto-$port";<br />
$id =~ s/\.//g;<br />
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";<br />
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";<br />
print "$sql\n" if $dry;<br />
print SQL "$sql;\n";<br />
<br />
# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");<br />
}<br />
}<br />
}<br />
}<br />
<br />
# break inserts into 100 records at a time<br />
&debug("inserting $#values +1 values");<br />
while ($#values > 0) {<br />
my $sql = "insert into portTotals_$classC values ";<br />
my $max_index = $#values > 100 ? 100 : $#values;<br />
for (my $i=0; $i<=$max_index; $i++) {<br />
$sql .= shift @values;<br />
$sql .= ',';<br />
}<br />
chop $sql;<br />
print "$sql\n" if $dry;<br />
print SQL "$sql;\n";<br />
}<br />
}<br />
<br />
# 12 1 8 1 1= 23<br />
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction<br />
# 12 1 8 1 1 3=26<br />
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto<br />
# 12 1 8 1 1 5=28<br />
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port<br />
#print "finished at ";<br />
#print `date`;<br />
`mv $queuedir/$file $archivedir` unless $dry;<br />
close(SQL);<br />
`bzip2 $sqldirworking/$outfile`;<br />
`mv $sqldirworking/$outfile.bz2 $sqldir/`;<br />
}<br />
}<br />
`rm $queuedir/.lock` unless $dry;<br />
<br />
sub debug {<br />
my $message = shift;<br />
if ($debug) {<br />
print "$message\n";<br />
}<br />
}<br />
<br />
# var full during ft-v05.2005-03-28.084500-0800 and<br />
# 2005-02-24 69.55.226<br />
<br />
# all port/daily totals before 2005-04-07<br />
</pre><br />
<br />
This script sends the sql files to the traffic server for processing:<br />
<pre>cat > /usr/home/flowbin/sendsql.pl<br />
#!/usr/bin/perl<br />
<br />
#use strict;<br />
#$debug=1;<br />
#$dry=1;<br />
<br />
my $remote = "69.55.233.199";<br />
my $sqldir = "/usr/home/sql";<br />
my $archive = "/usr/home/archive";<br />
my $sqldirremote = "/data/bwdb2/pending/";<br />
my @err;<br />
unless ($dry) {<br />
if (-e "$sqldir/.lock") {<br />
open(FILE, "$sqldir/.lock");<br />
my $pid = <FILE>;<br />
chomp($pid);<br />
close(FILE);<br />
if (kill(0, $pid)) {<br />
#another process is using the queue, bail out<br />
exit(0);<br />
}<br />
else {<br />
#dead lock file, remove it<br />
`rm $sqldir/.lock`;<br />
}<br />
}<br />
open(FILE, "> $sqldir/.lock");<br />
print FILE "$$\n";<br />
close(FILE);<br />
}<br />
<br />
opendir(DIR, $sqldir);<br />
my @files = readdir(DIR);<br />
closedir(DIR);<br />
<br />
foreach my $file (sort @files) {<br />
next unless $file =~ /bz2$/;<br />
<br />
my $r = `scp -Cq $sqldir/$file $remote:$sqldirremote 2>&1`;<br />
# print "scp $sqldir/$file $remote:$sqldirremote";<br />
unless ($?==0) {<br />
push @err, "scp -Cq $sqldir/$file $remote:$sqldirremote ($r)";<br />
}<br />
else {<br />
`mv $sqldir/$file $archive`;<br />
`ssh $remote mv $sqldirremote/$file $sqldirremote/${file}.done`;<br />
}<br />
}<br />
<br />
`rm $sqldir/.lock` unless $dry;<br />
<br />
if (@err) {<br />
email_support('bwdb2: sendsql.pl error',join "\n", @err);<br />
}<br />
<br />
sub email_support {<br />
my $subj=shift;<br />
my $body=shift;<br />
use Mail::Sendmail;<br />
<br />
# prepare message<br />
my %mail = (<br />
To => 'support@johncompanies.com,dave@johncompanies.com',<br />
From => 'support@johncompanies.com',<br />
Subject => $subj,<br />
Message => $body,<br />
smtp => 'mail.johncompanies.com',<br />
);<br />
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";<br />
}<br />
<br />
sub debug {<br />
my $message = shift;<br />
if ($debug) {<br />
print "$message\n";<br />
}<br />
}<br />
<br />
# var full during ft-v05.2005-03-28.084500-0800 and<br />
# 2005-02-24 69.55.226<br />
<br />
# all port/daily totals before 2005-04-07<br />
</pre><br />
<br />
<pre>crontab -e<br />
#process flows<br />
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl<br />
#move sql commands to traffic db<br />
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl<br />
</pre><br />
<br />
==== flow processing: castle ====<br />
<pre><br />
cat > /usr/home/flowbin/processflows.pl<br />
<br />
#!/usr/bin/perl<br />
<br />
#use strict;<br />
#$debug=1;<br />
#$dry=1;<br />
<br />
my $log = '/usr/home/flowbin/discards.log';<br />
<br />
use Data::Dumper;<br />
<br />
BEGIN {<br />
push @INC, "/usr/home/flowbin";<br />
}<br />
<br />
use db;<br />
<br />
#my $queuedir = "/usr/home/queue";<br />
my $queuedir = "/usr/home/working";<br />
my $archivedir = "/usr/home/archive";<br />
<br />
unless ($dry) {<br />
if (-e "$queuedir/.lock") {<br />
open(FILE, "$queuedir/.lock");<br />
my $pid = <FILE>;<br />
chomp($pid);<br />
close(FILE);<br />
if (kill(0, $pid)) {<br />
#another process is using the queue, bail out<br />
exit(0);<br />
}<br />
else {<br />
#dead lock file, remove it<br />
`rm $queuedir/.lock`;<br />
}<br />
}<br />
open(FILE, "> $queuedir/.lock");<br />
print FILE "$$\n";<br />
close(FILE);<br />
}<br />
<br />
my $db = db->new();<br />
$db->connect('traffic', '', 'root', '5over3') || die $db->{'error'};<br />
<br />
opendir(DIR, $queuedir);<br />
my @files = readdir(DIR);<br />
closedir(DIR);<br />
<br />
foreach my $file (sort @files) {<br />
unless($file =~ /^\./) {<br />
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;<br />
my $date = "$1 $2:$3:$4";<br />
my $condensedDate = $1;<br />
$condensedDate =~ s/-//g;<br />
my $iptotal = {};<br />
my $protototal = {};<br />
my $porttotal = {};<br />
<br />
&debug("started file $file at ");<br />
&debug(`date`);<br />
&debug("getting raw flow data (flow-print)");<br />
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;<br />
&debug("aggregating data at ");<br />
&debug(`date`);<br />
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }<br />
LOOP: while (my $line = readline DATA) {<br />
my @d = split /[\s]+/, $line;<br />
if ($d[0] ne '' && $d[0] ne 'Start') {<br />
my $addr = 0;<br />
my $port = 0;<br />
<br />
#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets<br />
#0 1 2 3 4 5 6 7 8 9 10 11<br />
#|<br />
# outbound = 2, inbound = 1<br />
<br />
my (@src_ip) = split '\.', $d[3];<br />
my (@dst_ip) = split '\.', $d[6];<br />
<br />
if ($src_ip[0] == 69 && $src_ip[1] == 55 &&<br />
$src_ip[2] >= 224 && $src_ip[2] <= 239 &&<br />
$src_ip[2] != 229 && $src_ip[2] != 231) { # for castle<br />
# if ($src_ip[0] == 69 && $src_ip[1] == 55 && $src_ip[2] == 229) { # for i2b<br />
$d[2] = 2;<br />
# hack for outbound bulk traffic counted 2x<br />
if ($dst_ip[2] == 234) { $d[11] /= 2; $d[10] /= 2; }<br />
}<br />
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 &&<br />
$dst_ip[2] >= 224 && $dst_ip[2] <= 239 &&<br />
$dst_ip[2] != 229 && $dst_ip[2] != 231) { # for castle<br />
# elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && $dst_ip[2] == 229) { # for i2b<br />
$d[2] = 1;<br />
}<br />
else { next LOOP; }<br />
<br />
if ($d[2] == 2) {<br />
$addr = $d[3];<br />
# if the dst-port is low, store that<br />
if ($d[7] <= 1024) { $port = $d[7]; }<br />
# if the src-port is low, store that<br />
elsif ($d[4] <= 1024) { $port = $d[4]; }<br />
else { $port = 99999; }<br />
}<br />
elsif ($d[2] == 1) {<br />
$addr = $d[6];<br />
# if the dst-port is high, assume its return traffic, try to store src-port if low<br />
if ($d[7] > 1024) {<br />
if ($d[4] <= 1024) { $port = $d[4]; }<br />
else { $port = 99999; }<br />
} else {<br />
$port = $d[7];<br />
}<br />
} else {<br />
next LOOP;<br />
}<br />
<br />
my (@ip) = split '\.', $addr;<br />
unless ($ip[0] == 69) { next LOOP; }<br />
unless ($ip[1] == 55) { next LOOP; }<br />
unless ($ip[2] >= 224 && $ip[2] <= 239 && $ip[2] != 229 && $ip[2] != 231) { next LOOP; }<br />
# unless ($ip[2] == 229) { next LOOP; }<br />
<br />
my $classC = "$ip[0]_$ip[1]_$ip[2]";<br />
<br />
# IP dir<br />
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }<br />
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }<br />
#<br />
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }<br />
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }<br />
#<br />
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }<br />
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }<br />
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];<br />
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];<br />
<br />
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];<br />
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];<br />
<br />
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];<br />
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];<br />
}<br />
}<br />
close(DATA);<br />
`rm /usr/home/working/tmp-$file`;<br />
&debug("processing ip totals at ");<br />
&debug(`date`);<br />
foreach my $classC (keys(%{$iptotal})) {<br />
$db->query("lock tables dailyIpTotals_$classC write") unless $dry;<br />
my @values;<br />
foreach my $ip (keys(%{$iptotal->{$classC}})) {<br />
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {<br />
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};<br />
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};<br />
# $packets = $packets > 2147483647 ? 0 : $packets;<br />
if ($octets > 2147483647) {<br />
my $ddir = $dir==1 ? 'in' : 'out';<br />
`echo "$date $ip $ddir $octets\n" >> $log`;<br />
# $octets = 0;<br />
}<br />
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction<br />
my $id = "$ip-$condensedDate-$dir";<br />
$id =~ s/\.//g;<br />
push @values, "('$date', '$ip', $dir, $octets, $packets)";<br />
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";<br />
print "$sql\n" if $dry;<br />
$db->query($sql) unless $dry;<br />
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");<br />
}<br />
}<br />
$db->query("unlock tables") unless $dry;<br />
<br />
$db->query("lock tables ipTotals_$classC write") unless $dry;<br />
# break inserts into 100 records at a time<br />
&debug("inserting $#values +1 values");<br />
while ($#values > 0) {<br />
my $sql = "insert into ipTotals_$classC values ";<br />
my $max_index = $#values > 100 ? 100 : $#values;<br />
for (my $i=0; $i<=$max_index; $i++) {<br />
$sql .= shift @values;<br />
$sql .= ',';<br />
}<br />
chop $sql;<br />
print "$sql\n" if $dry;<br />
$db->query($sql) unless $dry;<br />
}<br />
$db->query("unlock tables") unless $dry;<br />
}<br />
<br />
sleep 20;<br />
# &debug("processing protocol totals at ");<br />
# &debug(`date`);<br />
# foreach my $classC (keys(%{$protototal})) {<br />
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;<br />
# my @values;<br />
# foreach my $ip (keys(%{$protototal->{$classC}})) {<br />
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {<br />
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {<br />
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};<br />
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};<br />
# # $octets = $octets > 2147483647 ? 0 : $octets;<br />
# # $packets = $packets > 2147483647 ? 0 : $packets;<br />
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto<br />
# my $id = "$ip-$condensedDate-$dir-$proto";<br />
# $id =~ s/\.//g;<br />
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";<br />
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";<br />
# print "$sql\n" if $dry;<br />
# $db->query($sql) unless $dry;<br />
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");<br />
# }<br />
# }<br />
# }<br />
# $db->query("unlock tables") unless $dry;<br />
# my $sql = "insert into protoTotals_$classC values ";<br />
# $sql .= join ',', @values;<br />
# $db->query("lock tables protoTotals_$classC write") unless $dry;<br />
# print "$sql\n" if $dry;<br />
# $db->query($sql) unless $dry;<br />
# $db->query("unlock tables") unless $dry;<br />
# }<br />
<br />
&debug("processing port totals at ");<br />
&debug(`date`);<br />
foreach my $classC (keys(%{$porttotal})) {<br />
$db->query("lock tables dailyPortTotals_$classC write") unless $dry;<br />
my @values;<br />
foreach my $ip (keys(%{$porttotal->{$classC}})) {<br />
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {<br />
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {<br />
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {<br />
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};<br />
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};<br />
# $octets = $octets > 2147483647 ? 0 : $octets;<br />
# $packets = $packets > 2147483647 ? 0 : $packets;<br />
<br />
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port<br />
my $id = "$ip-$condensedDate-$dir-$proto-$port";<br />
$id =~ s/\.//g;<br />
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";<br />
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";<br />
print "$sql\n" if $dry;<br />
$db->query($sql) unless $dry;<br />
# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");<br />
}<br />
}<br />
}<br />
}<br />
$db->query("unlock tables") unless $dry;<br />
<br />
$db->query("lock tables portTotals_$classC write") unless $dry;<br />
# break inserts into 100 records at a time<br />
&debug("inserting $#values +1 values");<br />
while ($#values > 0) {<br />
my $sql = "insert into portTotals_$classC values ";<br />
my $max_index = $#values > 100 ? 100 : $#values;<br />
for (my $i=0; $i<=$max_index; $i++) {<br />
$sql .= shift @values;<br />
$sql .= ',';<br />
}<br />
chop $sql;<br />
print "$sql\n" if $dry;<br />
$db->query($sql) unless $dry;<br />
}<br />
$db->query("unlock tables") unless $dry;<br />
sleep 10;<br />
}<br />
<br />
# 12 1 8 1 1= 23<br />
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction<br />
# 12 1 8 1 1 3=26<br />
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto<br />
# 12 1 8 1 1 5=28<br />
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port<br />
#print "finished at ";<br />
#print `date`;<br />
`mv $queuedir/$file $archivedir` unless $dry;<br />
}<br />
}<br />
`rm $queuedir/.lock` unless $dry;<br />
<br />
sub debug {<br />
my $message = shift;<br />
if ($debug) {<br />
print "$message\n";<br />
}<br />
}<br />
<br />
# var full during ft-v05.2005-03-28.084500-0800 and<br />
# 2005-02-24 69.55.226<br />
<br />
# all port/daily totals before 2005-04-07<br />
</pre><br />
<br />
<pre><br />
cat > /usr/home/flowbin/db.pm<br />
<br />
#!/usr/bin/perl<br />
#<br />
# $Header: /usr/cvs/newgw/lib/db.pm,v 1.4 2003/06/05 18:20:01 glenn Exp $<br />
#<br />
# Copyright (c) 2003<br />
# e-Monitoring Networks, Inc. All rights reserved.<br />
#<br />
#<br />
package db;<br />
<br />
use strict;<br />
use DBI;<br />
<br />
sub new {<br />
my $class = shift;<br />
my $self = {};<br />
<br />
$self->{'debug'} = 0;<br />
bless $self, $class;<br />
<br />
return $self;<br />
}<br />
<br />
sub connect {<br />
my $self = shift;<br />
my $dbname = shift;<br />
my $dbhost = shift;<br />
my $dbuser = shift;<br />
my $dbpass = shift;<br />
<br />
my $host = '';<br />
if (defined($dbhost)) {<br />
$host = ";host=$dbhost";<br />
}<br />
<br />
eval {<br />
$self->debug("connecting to: DBI:mysql:database=$dbname;$host", 1);<br />
$self->{'dbh'} = DBI->connect("DBI:mysql:database=$dbname;$host", $dbuser, $dbpass);<br />
};<br />
if ($self->{'dbh'}) {<br />
return 1;<br />
}<br />
$self->{'error'} = "Error connecting to database $@";<br />
$self->debug("Error connecting to database $@");<br />
return 0;<br />
}<br />
<br />
sub query {<br />
my $self = shift;<br />
my $query = shift;<br />
<br />
$self->debug($query, 1);<br />
my $sth;<br />
eval {<br />
$sth = $self->{'dbh'}->prepare($query);<br />
};<br />
unless ($sth) {<br />
$self->{'error'} = "error preparing query $@";<br />
$self->debug("error preparing query $@");<br />
return undef;<br />
}<br />
my $qty;<br />
eval {<br />
$qty = $sth->execute;<br />
};<br />
unless ($qty) {<br />
$self->{'error'} = "error executing query $@";<br />
warn "error executing query $@ $query";<br />
return undef;<br />
}<br />
$self->debug("returning $qty, $sth from query", 6);<br />
return ($qty, $sth);<br />
}<br />
<br />
sub disconnect {<br />
my $self = shift;<br />
<br />
$self->{'dbh'}->disconnect;<br />
return 0;<br />
}<br />
<br />
sub debug {<br />
my $self = shift;<br />
my $msg = shift;<br />
my $level = shift || 0;<br />
<br />
if ($level < $self->{'debug'}) {<br />
print "$msg\n";<br />
}<br />
return 0;<br />
}<br />
1;<br />
</pre><br />
<br />
mkdir /usr/home/archive<br />
mkdir -p /usr/home/sql/tmp<br />
<br />
<pre>crontab -e<br />
#process flows<br />
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl</pre><br />
<br />
==== setup traffic db ====<br />
* Install mysql:<br />
<pre>cd /usr/ports/databases/mysql50-server<br />
make install clean</pre><br />
<br />
cat >> /etc/rc.conf<br />
mysql_enable="YES" <br />
<br />
Move db data dir:<br />
/usr/local/etc/rc.d/mysql-server stop<br />
mkdir /usr/home/database/<br />
mv /var/db/mysql/* /usr/home/database/<br />
chown -R mysql:mysql /usr/home/database<br />
<br />
Edit database location in startup script:<br />
vi /usr/local/etc/rc.d/mysql-server<br />
# : ${mysql_dbdir="/var/db/mysql"}<br />
: ${mysql_dbdir="/usr/home/database"}<br />
<br />
/usr/local/etc/rc.d/mysql-server start<br />
<br />
<br />
* Install mysql perl database modules:<br />
<pre><br />
cd /usr/ports/databases/p5-DBI<br />
make install clean<br />
cd /usr/ports/databases/p5-DBD-mysql50<br />
make install clean<br />
(no to SSL support)<br />
</pre><br />
<br />
* Setting up database<br />
<pre><br />
rehash<br />
/usr/local/etc/rc.d/mysql-server start<br />
mysql -u root<br />
create database traffic;<br />
grant all on *.* to root@localhost identified by '5over3';<br />
grant all on traffic.* to jc@10.1.4.5 identified by '2gMKY3Wt';<br />
<br />
</pre><br />
<br />
If this was a new server we'd setup new tables. See [[#mysql_2|mysql]] for how those tables would be setup.<br />
<br />
We are assuming here we are moving data from an existing db, here's how that's done (from the current traffic db):<br />
rsync -av --progress /usr/home/database/traffic/ 10.1.4.203:/usr/home/database/traffic/<br />
<br />
When you're ready to do the cutover, shut down mysql on both hosts and do one last sync.<br />
<br />
<br />
<br />
==== process flows from bwdb2 ====<br />
On traffic database server (bwdb):<br />
<br />
<pre>crontab -e<br />
#import sql from bwdb2<br />
10,25,40,55 * * * * /usr/home/flowbin/processsql.pl</pre><br />
<br />
Add access to mysql: <br />
<pre>mysql -u root -p <br />
grant all on traffic.* to bwdb2@localhost identified by 's1lver4d';<br />
</pre> <br />
<br />
<pre>cat > /usr/home/flowbin/processsql.pl<br />
<br />
#!/usr/bin/perl<br />
<br />
#use strict;<br />
#$debug=1;<br />
#$dry=1;<br />
<br />
my $sqldir = "/usr/home/bwdb2/pending";<br />
my $mysql = '/usr/local/bin/mysql';<br />
my @err;<br />
unless ($dry) {<br />
if (-e "$sqldir/.lock") {<br />
open(FILE, "$sqldir/.lock");<br />
my $pid = <FILE>;<br />
chomp($pid);<br />
close(FILE);<br />
if (kill(0, $pid)) {<br />
#another process is using the queue, bail out<br />
exit(0);<br />
}<br />
else {<br />
#dead lock file, remove it<br />
`rm $sqldir/.lock`;<br />
}<br />
}<br />
open(FILE, "> $sqldir/.lock");<br />
print FILE "$$\n";<br />
close(FILE);<br />
}<br />
<br />
opendir(DIR, $sqldir);<br />
my @files = readdir(DIR);<br />
closedir(DIR);<br />
<br />
foreach my $file (sort @files) {<br />
next unless $file =~ /done$/;<br />
my $r = `bzcat $sqldir/$file | $mysql -u bwdb2 -ps1lver4d traffic`;<br />
unless ($?==0) {<br />
push @err, "bzcat $sqldir/$file | $mysql -u bwdb2 -pxxxxx traffic ($r)";<br />
}<br />
else {<br />
`rm $sqldir/$file`;<br />
}<br />
}<br />
<br />
`rm $sqldir/.lock` unless $dry;<br />
<br />
if (@err) {<br />
email_support('bwdb: processsql.pl error',join "\n", @err);<br />
}<br />
<br />
sub email_support {<br />
my $subj=shift;<br />
my $body=shift;<br />
use Mail::Sendmail;<br />
<br />
# prepare message<br />
my %mail = (<br />
To => 'dave@johncompanies.com',<br />
From => 'support@johncompanies.com',<br />
Subject => $subj,<br />
Message => $body,<br />
smtp => 'mail.johncompanies.com',<br />
);<br />
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";<br />
}<br />
<br />
sub debug {<br />
my $message = shift;<br />
if ($debug) {<br />
print "$message\n";<br />
}<br />
}<br />
</pre><br />
<br />
chmod 0700 /usr/home/flowbin/processsql.pl<br />
<br />
Make sure bwdb is reachable from the outside only to bwdb2:<br />
<br />
On nat, add to <tt>/etc/ipnat.rules</tt><br />
<pre># bwdb<br />
bimap fxp0 10.1.4.203/32 -> 69.55.233.199/32</pre><br />
<br />
Reload:<br />
ipnat -C -F -f /etc/ipnat.rules<br />
<br />
Setup firewall rule on firewall:<br />
ipfw add 00094 allow ip from 66.181.18.5 to 69.55.233.199 22<br />
ipfw add 00094 deny ip from any to 69.55.233.199<br />
<br />
Setup firewall on bwdb to restrict access now that it's nat'd:<br />
<pre><br />
cat >> /usr/local/etc/rc.d/boot.sh<br />
ipfw add 1 allow tcp from any to any established<br />
ipfw add 2 allow ip from 10.1.4.0/24,66.181.18.5,69.55.233.195 to me 22<br />
ipfw add 3 allow ip from 10.1.4.5 to me 3306<br />
ipfw add 4 allow ip from 69.55.225.225 53 to me <br />
ipfw add 5 allow ip from 69.55.230.2 25 to me <br />
ipfw add 6 allow ip from me to me 4444<br />
ipfw add 7 allow icmp from any to me<br />
ipfw add 8 allow udp from 10.1.4.203 to 10.1.4.203 dst-port 4444<br />
ipfw add 9 allow udp from 10.1.4.5 to me 161<br />
ipfw add 100 deny ip from any to me<br />
</pre> <br />
<br />
chmod 0700 /usr/local/etc/rc.d/boot.sh<br />
<br />
From bwdb2, add ssh key:<br />
cat /root/.ssh/id_dsa.pub | ssh 69.55.233.199 'cat - >> /root/.ssh/authorized_keys' <br />
<br />
Confirm no password access:<br />
ssh 69.55.233.199 hostname<br />
<br />
= bwdb2 =<br />
== Summary ==<br />
This machine tracks and stores network traffic (netflow) at i2b. It is our means to monitor customer bandwidth usage.<br />
<br />
* Location: i2b, cab6<br />
* OS: FreeBSD 6.4 x86<br />
* Networking: Priv IP: 10.1.2.4 There are 2 onboard nic's, one of which is the "listener"<br />
* Hardware: Custom 2U. Single power supply. <br />
* Drives: two 150 GB (2 x 150GB) RAID1 arrays running on a 3ware 7006 RAID card.<br />
<br />
== Services Provided ==<br />
* netflow<br />
* bigbrother <br />
<br />
== netflow ==<br />
<br />
The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.<br />
<br />
A cronjob moves that flow file (or files if there are multiple due to some delay) <br />
1,16,31,46 * * * * /usr/home/flowbin/queue.pl<br />
<br />
into a processing queue:<br />
<tt>/usr/home/working</tt><br />
<br />
Then a separate file processes whatever flow files it finds there, and builds sql files ready for insertion into the traffic database:<br />
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl<br />
<br />
Then yet another process copies the sql files to the traffic database server for processing and insertion into the mysql database:<br />
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl<br />
<br />
== Regular maintenance ==<br />
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]<br />
<br />
* if space becomes tight, move sql files and flow files to backup server, both located in <tt>/usr/home/flowbin/archive</tt><br />
<br />
= firewall (newgateway) =<br />
<br />
== Summary ==<br />
<br />
This machine is the primary (only) firewall for the entire network at castle. <br />
<br />
* Location: castle, cab 3-8<br />
* OS: FreeBSD 4.11 x86<br />
* Networking: Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).<br />
* Hardware: 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card.<br />
<br />
== Services Provided ==<br />
* firewall (ipfw)<br />
* snmp<br />
* bigbrother<br />
<br />
== Firewall Rule Configuration ==<br />
<br />
See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.<br />
<br />
== Disaster Recovery ==<br />
<br />
If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.<br />
<br />
Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)<br />
<br />
<pre>hostname="newgateway.johncompanies.com"<br />
firewall_script="/etc/firewall.sh"<br />
firewall_enable="NO"<br />
sendmail_enable="NONE"<br />
sshd_enable="YES"<br />
inetd_enable="NO"<br />
xntpd_enable="YES"<br />
snmpd_enable="YES"<br />
#snmpd_flags="-as -p /var/run/snmpd.pid"<br />
#ipnat_enable="YES"<br />
#ipnat_rules="/etc/ipnat.rules"<br />
gateway_enable="YES"<br />
<br />
defaultrouter="69.55.233.161"<br />
<br />
ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0"<br />
ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248"<br />
<br />
#<br />
# Original JohnCompanies 69.55.224.0/20<br />
#<br />
ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248"<br />
<br />
static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1<br />
2 route13 route14 route15 route16 route17 route18"<br />
<br />
route_route1="-net 69.55.224.0 69.55.233.153"<br />
route_route2="-net 69.55.225.0 69.55.233.153"<br />
route_route3="-net 69.55.226.0 69.55.233.153"<br />
route_route4="-net 69.55.227.0 69.55.233.153"<br />
route_route5="-net 69.55.228.0 69.55.233.153"<br />
route_route6="-net 69.55.229.0 69.55.233.153"<br />
route_route7="-net 69.55.230.0 69.55.233.153"<br />
route_route8="-net 69.55.231.0 69.55.233.153"<br />
route_route9="-net 69.55.232.0 69.55.233.153"<br />
route_route10="-net 69.55.233.0 69.55.233.153"<br />
route_route11="-net 69.55.234.0 69.55.233.153"<br />
route_route12="-net 69.55.235.0 69.55.233.153"<br />
route_route13="-net 69.55.236.0 69.55.233.153"<br />
route_route14="-net 69.55.237.0 69.55.233.153"<br />
route_route15="-net 69.55.238.0 69.55.233.153"<br />
route_route16="-net 69.55.239.0 69.55.233.153"<br />
route_route17="-net 10.1.5.0 10.1.4.2"<br />
route_route18="-net 10.1.6.0 10.1.4.2"<br />
<br />
<br />
#In case of 3750 failure:<br />
#defaultrouter="69.43.128.81"<br />
#ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248"<br />
<br />
#bind .1's here:<br />
#ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0"<br />
<br />
#bulk:<br />
# reassign 69.55.231.1 to the int iface on the firewall<br />
# set the DG on the firewall to 69.43.138.9<br />
# set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248</pre><br />
<br />
== Cronjobs ==<br />
1 0 * * * /usr/local/etc/rsync.backup<br />
Backup to backup1<br />
<br />
0 0 1 * * /sbin/ipfw zero<br />
0 0 1 * * /sbin/ipfw del 3 4 5 17331<br />
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap). <br />
<br />
Inside <tt>/etc/daily.local</tt> you will see a call to <tt>/etc/makepiperules.pl</tt><br />
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.<br />
<br />
== DOS attacks ==<br />
<br />
See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.<br />
<br />
Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will<br />
# enter a rule that allows all UDP to go through<br />
# send an emergency email to support and indicating an attack is in progress<br />
# send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted<br />
# wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1<br />
This file lives under /usr/home/user/doswatch.pl<br />
To run:<br />
cd /usr/home/user<br />
./doswatch.pl &<br />
<br />
To kill;<br />
fg<br />
^C<br />
<br />
It writes its findings to /usr/home/user/doswatch.log<br />
<br />
= backup1 =<br />
<br />
== Summary ==<br />
<br />
This machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server. <br />
<br />
* Location: castle, cab 3-8<br />
* OS: Ubuntu 8.04.1 server x86<br />
* Networking: Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)<br />
* Hardware: 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Single power supply. <br />
* Drives: 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card<br />
<br />
== Services provided ==<br />
* backup via rsync<br />
* mysql - traffic data<br />
* nfs server - for backups<br />
* snmp client - for big brother<br />
* bigbrother client<br />
<br />
== Usage and Notes ==<br />
* all data is stored under /data<br />
* virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: <tt>/backup1</tt>), and they all have their ssh keys setup to allow passwordless rsync's<br />
* each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)<br />
* at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.<br />
* cancelled customer systems are compressed and stored under <tt>/data/deprecated</tt><br />
* archived bwdb2 flow files are stored under <tt>/data/bwdb2</tt><br />
* critical files from backup2 are stored under <tt>/data/backup2</tt><br />
<br />
== Cronjobs ==<br />
<pre><br />
00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null<br />
35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate /data/backuplog.log</pre><br />
this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup. It will email support@johncompanies.com at it’s conclusion. This email can be deleted, however note when it begins to take significantly longer to complete, ie runs past 2200 pm – this usually indicates a problem on the backup server.<br />
<br />
<pre>10,25,40,55 * * * * /usr/local/sbin/processsql.pl<br />
</pre><br />
this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.<br />
<pre>0 0 * * * /usr/local/sbin/3wraidchk<br />
</pre><br />
checks the health of the RAID array<br />
<br />
== Regular maintenance ==<br />
*[[Routine_Maintenance#Free_up_space_on_backup1|Remove old backups]]<br />
*[[Routine_Maintenance#3ware|Check on auto-verify]]<br />
<br />
== build ==<br />
<br />
<pre>Setup raid5 with a boot vol of 12G 5.45tb <br />
12G boot<br />
4664 GB <br />
<br />
Install ubuntu 8.04<br />
<br />
Swap 4G<br />
<br />
Don’t format data drive<br />
<br />
http://www.unixgods.org/~tilo/linux_larger_2TB.html<br />
<br />
parted /dev/sdb<br />
print<br />
mklabel gpt<br />
print<br />
<br />
#Disk /dev/sdb: 4987GB<br />
#Sector size (logical/physical): 512B/512B<br />
#Partition Table: gpt<br />
<br />
#Number Start End Size File system Name Flags<br />
<br />
mkpart primary ext3 0 4987GB<br />
print<br />
<br />
#Disk /dev/sdb: 5987GB<br />
#Sector size (logical/physical): 512B/512B<br />
#Partition Table: gpt<br />
<br />
#Number Start End Size File system Name Flags<br />
# 1 17.4kB 4987GB 4987GB primary<br />
<br />
quit<br />
<br />
mkfs.ext3 /dev/sdb1<br />
#mke2fs 1.40.8 (13-Mar-2008)<br />
#Filesystem label=<br />
#OS type: Linux<br />
#Block size=4096 (log=2)<br />
#Fragment size=4096 (log=2)<br />
#304390144 inodes, 1217544183 blocks<br />
#60877209 blocks (5.00%) reserved for the super user<br />
#First data block=0<br />
#Maximum filesystem blocks=0<br />
#37157 block groups<br />
#32768 blocks per group, 32768 fragments per group<br />
#8192 inodes per group<br />
#Superblock backups stored on blocks:<br />
# 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,<br />
# 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,<br />
# 102400000, 214990848, 512000000, 550731776, 644972544<br />
#<br />
#Writing inode tables: 967/37157<br />
<br />
mkdir /data<br />
<br />
#root@backup1:~# df -h<br />
#Filesystem Size Used Avail Use% Mounted on<br />
#/dev/sda2 8.3G 540M 7.3G 7% /<br />
#varrun 1013M 40K 1013M 1% /var/run<br />
#varlock 1013M 0 1013M 0% /var/lock<br />
#udev 1013M 56K 1013M 1% /dev<br />
#devshm 1013M 0 1013M 0% /dev/shm<br />
#/dev/sdb1 4.5T 192M 4.3T 1% /data<br />
<br />
<br />
apt-get update<br />
apt-get upgrade<br />
apt-get install snmp snmpd ntp nfs-kernel-server<br />
<br />
echo "\"\e[5~\": history-search-backward" >> ~/.inputrc<br />
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc<br />
<br />
vi /etc/ntp.conf<br />
server 10.1.4.5<br />
<br />
scp root@10.1.4.3:/root/.ssh/authorized_keys /root/.ssh/<br />
cd /root/<br />
ssh-keygen -t dsa<br />
echo "10.1.4.3 backup2" >> /etc/hosts<br />
<br />
cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' <br />
<br />
ssh backup2<br />
<br />
vi /root/.bashrc<br />
export PS1="[\u@\h \w]# "<br />
alias h='history'<br />
alias vi='vim'<br />
alias j='jobs'<br />
export PS1="[\u@\h \w]# "<br />
alias dr='screen -dr'<br />
export EDITOR=vim<br />
export GREP_OPTIONS='--color=auto'<br />
export HISTFILESIZE=1000<br />
<br />
source /root/.bashrc<br />
<br />
echo "# ttyS0 - getty <br />
# <br />
# This service maintains a getty on ttyS0 from the point the system is<br />
# started until it is shut down again.<br />
<br />
start on runlevel 2<br />
start on runlevel 3<br />
start on runlevel 4<br />
start on runlevel 5<br />
<br />
stop on runlevel 0<br />
stop on runlevel 1<br />
stop on runlevel 6<br />
<br />
respawn<br />
exec /sbin/getty 38400 ttyS0" > /etc/event.d/ttyS0 <br />
<br />
<br />
vi /boot/grub/menu.lst<br />
<br />
serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1<br />
terminal --timeout=15 serial console<br />
<br />
append to kernel lines: <br />
console=tty0 console=ttyS0,38400n8<br />
<br />
show menu:<br />
#hiddenmenu<br />
<br />
echo 'rocommunity jcread 10.1.4.5<br />
rocommunity jcread 10.1.4.3<br />
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf<br />
<br />
# to see which iface it is, on backup2:<br />
<br />
snmpwalk -v 1 -c jcread 10.1.4.8 interface<br />
<br />
<br />
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd<br />
<br />
echo "bb:x:1984:" >> /etc/group<br />
<br />
pwconv<br />
<br />
mkdir /home/bb<br />
chown bb.bb /home/bb<br />
<br />
cd ~bb<br />
scp backup2:/mnt/data4/build/bb/bb-linux.tar .<br />
<br />
tar xf bb-linux.tar<br />
<br />
cd /home/bb/bbc1.9e-btf/etc<br />
<br />
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts<br />
echo "10.1.4.8 backup1.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts<br />
<br />
echo "/:90:95<br />
/var:90:95<br />
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab<br />
<br />
<br />
vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh<br />
(remove all | SORT xxxx)<br />
<br />
chmod +r /var/log/messages<br />
<br />
./bbchkcfg.sh <br />
#(y to questions)<br />
./bbchkhosts.sh<br />
#(ignore ssh errors)<br />
cd ../..<br />
chown -R bb .<br />
su bb<br />
cd<br />
cd bbc1.9e-btf/src<br />
<br />
#make; make install<br />
cd ..<br />
./runbb.sh start<br />
more BBOUT <br />
(look for errors)<br />
exit<br />
<br />
vi /etc/rc.local<br />
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"<br />
<br />
<br />
echo '/data 10.1.4.0/24(rw, no_root_squash,async,no_subtree_check)' >> /etc/exports<br />
<br />
/etc/init.d/nfs-kernel-server restart<br />
<br />
<br />
echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd<br />
<br />
<br />
echo '10.1.4.8 backup1' >> /etc/hosts<br />
echo '/dev/sdb1 /data ext3 rw,noatime 0 0' >> /etc/fstab<br />
<br />
<br />
to install digi drivers:<br />
<br />
wget http://ftp1.digi.com/support/driver/40002086_n.tgz<br />
apt-get install linux-image-2.6.24-19-server<br />
apt-get install linux-source-2.6.24 (not needed?)<br />
apt-get install linux-headers-2.6.24-19-server <br />
apt-get install make<br />
apt-get install gcc<br />
apt-get install g++ <br />
apt-get install libncurses5-dev<br />
apt-get install expect<br />
apt-get install libdbi-perl libdate-calc-perl libdbd-mysql-perl<br />
<br />
cd /usr/src; ln -s linux-headers-2.6.24-19-server linux<br />
./configure<br />
make all<br />
make install<br />
make postinstall<br />
<br />
/usr/bin/dgrp_cfg_node -v -v init el 65.116.11.2 8<br />
<br />
apt-get install mysql<br />
<br />
mkdir /data/mysql<br />
chown mysql:mysql /data/mysql<br />
/etc/init.d/mysql stop<br />
mv /var/lib/mysql/* /data/mysql/<br />
mv /data/mysql/ib_* /var/lib/mysql/<br />
vi /etc/mysql/my.cnf<br />
(change datadir to /data/mysql)<br />
vi /etc/apparmor.d/usr.sbin.mysqld<br />
add:<br />
/data/mysql/ r,<br />
/data/mysql/** rwk,<br />
Comment out:<br />
# /var/lib/mysql/ r,<br />
# /var/lib/mysql/** rwk,<br />
<br />
/etc/init.d/apparmor restart<br />
/etc/init.d/mysql start<br />
<br />
tw_cli /c0/u0 set ignoreECC=on<br />
tw_cli /c0/u0 set storsave=balance<br />
tw_cli /c0/u0 set cache=on<br />
<br />
<br />
0 0 * * * /usr/local/sbin/3wraidchk<br />
</pre><br />
<br />
= backup2 =<br />
<br />
== Summary ==<br />
<br />
This machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. NOTE: power button is broken, so the reset button (paper clip) was rewired to be the power button.<br />
<br />
* Location: castle, cab 3-7<br />
* OS: FreeBSD 6.1 x86<br />
* Networking: Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC infrastructure @ i2b)<br />
* Hardware: 16 IDE drive bays (4 columns of 4, drive 0-0 top left, drive 0-1 just to the right TODO) all hot-swap. Triple power supply. <br />
* Drives: <br />
**3ware 7500-8:<br />
***200 GB JBOD (1 x 200G) labeled 0-0<br />
***500 GB RAID5 (3 x 250G) 0-1 thru 0-3<br />
***700 GB RAID5 (4 x 250G) 0-4 thru 0-7<br />
**3ware 7500-8:<br />
***700 GB RAID5 (4 x 250G) 1-0 thru 1-3<br />
***700 GB RAID5 (4 x 250G) 1-4 thru 1-7<br />
<br />
All drives MUST be western digital IDE drives. Other brands will not fit.<br />
<br />
In case of an outage, nfs will hang on all connected servers until the nfs service returns. If you can't get backup2 back online, you can get nfs running elsewhere and fake backup2's MAC's: priv: 00:0e:0c:59:c1:a6, pub: 00:07:e9:5b:c6:45<br />
<br />
To configure:<br />
ifconfig fxp0 link 00:90:27:f9:0a:d9<br />
<br />
== Services provided ==<br />
* backup via rsync and nfs<br />
* samba<br />
* nfs<br />
* snmp<br />
* bigbrother<br />
<br />
== Usage ==<br />
* all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: <tt>/mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4</tt><br />
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/mnt/data2/iso</tt><br />
* this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data<br />
* this server serves as an archive for exported db data from bwdb and old flow files.<br />
* isys backs up here<br />
* customers are nfs-moutned under /mnt/data3/customers as file-backed md devices<br />
* in <tt>/mnt/data4</tt> there are lots of useful things used for building our vps servers, customer servers, and management scripts:<br />
** <tt>/bin</tt>: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)<br />
** <tt>/build</tt>: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on<br />
** <tt>/vzrpms</tt>: contains the OS templates for many-to-most of the OS's we offer on vz systems<br />
<br />
== Cronjobs ==<br />
* backs itself up nightly to nfs-mounted backup1 (mountpoint: <tt>/backup2</tt>)<br />
<br />
== Regular maintenance ==<br />
*[[Routine_Maintenance#3ware|Check on health]]<br />
<br />
= backup3 =<br />
== Summary ==<br />
This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups. <br />
<br />
* Location: i2b, cab 6<br />
* OS: Ubuntu 10.04.1 server amd64<br />
* Networking: Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2<br />
* Hardware: 16 drive SATA bays (4 columns of 4, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: 5 TB (6 x 1TB) RAID5 array running on an Areca Technology Corp. ARC-1160 16-Port<br />
<br />
== Services provided ==<br />
* backup via rsync and nfs<br />
* samba<br />
* nfs<br />
* digi realport<br />
* snmp<br />
* bigbrother<br />
<br />
== Usage ==<br />
* all data is stored under /data<br />
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt><br />
* this server serves as an archive for exported db data from bwdb and old flow files.<br />
* inftrastructure machines at i2b back up here<br />
* customers are nfs-moutned under /data/customers as file-backed loopback devices<br />
<br />
== management scripts ==<br />
* mkbackups<br />
<br />
mkbackup <cid> GB <ip><br />
<br />
== Cronjobs ==<br />
0 0 * * * /usr/local/sbin/arecaraidchk<br />
RAID checks<br />
<br />
35 4 * * * /usr/local/sbin/snapshot_archive<br />
Rotate daily snapshots for infrastructure machine backups<br />
<br />
== Regular maintenance ==<br />
*[[Routine_Maintenance#Areca|Check on RAID health]]<br />
<br />
== Build ==<br />
<br />
=== BIOS Config ===<br />
disable quiet boot<br />
<br />
set to last state after power loss<br />
<br />
set date/time to GMT<br />
<br />
enable serial console output (baud rate 115200)<br />
<br />
=== Install OS ===<br />
<pre>Ubuntu 10.04.1 amd64 (couldn't get 12.04 to load cause the H/W was incompat)<br />
10G / ext3<br />
2G swap<br />
~ /data ext4<br />
<br />
Install packages:<br />
openssh<br />
samba</pre><br />
<br />
=== DNS and private IP ===<br />
<br />
echo "nameserver 69.55.225.225" >> /etc/resolv.conf<br />
<br />
Add a 2nd IP to eth0 and setup priv net<br />
<pre>vi /etc/network/interfaces<br />
<br />
auto eth0<br />
iface eth0 inet static<br />
address 69.55.229.4<br />
netmask 255.255.255.0<br />
network 69.55.229.0<br />
broadcast 69.55.229.255<br />
gateway 69.55.229.1<br />
# dns-* options are implemented by the resolvconf package, if installed<br />
dns-nameservers 69.55.229.3 66.181.0.2<br />
dns-search johncompanies.com<br />
<br />
auto eth0:1<br />
iface eth0:1 inet static<br />
address 69.55.231.2<br />
netmask 255.255.255.0<br />
network 69.55.231.0<br />
broadcast 69.55.231.255<br />
<br />
auto eth1<br />
iface eth1 inet static<br />
address 10.1.2.3<br />
netmask 255.255.255.0<br />
network 10.1.2.0<br />
broadcast 10.1.2.255<br />
<br />
</pre><br />
<br />
=== Install packages ===<br />
<pre>apt-get update<br />
apt-get upgrade<br />
apt-get install gcc<br />
apt-get install libssl-dev<br />
apt-get install libncurses5-dev<br />
apt-get install cu<br />
apt-get install unzip<br />
apt-get install snmp snmpd ntp nfs-kernel-server</pre><br />
<br />
=== tweak grub, enable serial ===<br />
<br />
<pre>vi /etc/default/grub<br />
#GRUB_HIDDEN_TIMEOUT=0<br />
GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64"<br />
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"<br />
update-grub</pre><br />
<br />
<pre>echo "start on stopped rc RUNLEVEL=[2345]<br />
stop on runlevel [!2345]<br />
respawn<br />
exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf</pre><br />
<br />
=== install realport (digi) driver ===<br />
<br />
give the digi an ip with DgIpServ.exe<br />
<br />
<pre>cd /usr/src/<br />
wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz<br />
tar xzf dgrp-1.9.tgz <br />
cd dgrp-1.9/<br />
./configure<br />
make<br />
make install<br />
make postinstall<br />
update-rc.d dgrp_daemon defaults</pre><br />
<br />
configure ports:<br />
dgrp_cfg_node init el 10.1.2.10 16<br />
<br />
try connecting with:<br />
cu -l /dev/ttyel00 -s 38400<br />
<br />
=== shell, ntp, ssh key, hosts ===<br />
<br />
Shell autocompletion search:<br />
<pre>echo "\"\e[5~\": history-search-backward" >> ~/.inputrc<br />
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre><br />
<br />
Setup ntp:<br />
<pre>vi /etc/ntp.conf<br />
server 10.1.2.1<br />
server ntp.ubuntu.com</pre><br />
<br />
Generate ssh keys:<br />
<pre>cd /root/<br />
ssh-keygen -t dsa</pre><br />
Defaults, no password<br />
<br />
Setup hosts:<br />
<pre>echo "69.55.230.10 backup2" >> /etc/hosts<br />
echo "69.55.230.11 backup1" >> /etc/hosts<br />
echo "10.1.2.4 bwdb2" >> /etc/hosts<br />
echo "10.1.2.3 backup3" >> /etc/hosts</pre><br />
<br />
Copy keys to servers where we need passwordless login:<br />
<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' <br />
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre> <br />
<br />
Setup shell:<br />
<pre>vi /root/.bashrc<br />
(add to bottom)<br />
alias h='history'<br />
alias vi='vim'<br />
alias j='jobs'<br />
export PS1="[\u@\h \w]# "<br />
alias dr='screen -dr'<br />
export EDITOR=vim<br />
export GREP_OPTIONS='--color=auto'<br />
export HISTFILESIZE=1000<br />
<br />
alias tip-switch-p20='cu -l ttyel00 -s 9600'<br />
alias tip-switch-p21='cu -l ttyel15 -s 9600'<br />
alias tip-switch-p22='cu -l ttyel14 -s 9600'<br />
alias tip-switch-p23='cu -l ttyel05 -s 9600'<br />
alias tip-switch-p24='cu -l ttyel06 -s 9600'<br />
alias tip-switch-p25='cu -l ttyel09 -s 9600'<br />
alias tip-switch-p26='cu -l ttyel07 -s 9600'<br />
alias tip-switch-p27='cu -l ttyel08 -s 9600'<br />
alias tip-firewall2='cu -l ttyel01 -s 115200'<br />
alias tip-nat2='cu -l /dev/ttyel02 -s 115200'<br />
alias tip-backup3='cu -l ttyel04 -s 38400'<br />
alias tip-bwdb2='cu -l ttyel03 -s 115200'<br />
alias tip-backup4='cu -l ttyel13 -s 115200'<br />
alias tip-jail3='cu -l ttyel11 -s 115200'<br />
<br />
Load new shell:<br />
source /root/.bashrc<br />
<br />
Setup snmpd (this is only valid for a server at castle):<br />
echo 'rocommunity jcread 10.1.4.5<br />
rocommunity jcread 10.1.4.3<br />
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf<br />
<br />
to see which iface it is, on backup2:<br />
<br />
snmpwalk -v 1 -c jcread 10.1.4.8 interface<br />
<br />
=== nfs ===<br />
<br />
Allow mounts from private net:<br />
echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports<br />
<br />
Restart nfsd:<br />
/etc/init.d/nfs-kernel-server restart<br />
<br />
=== bb ===<br />
<br />
Add user, group:<br />
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd<br />
echo "bb:x:1984:" >> /etc/group<br />
pwconv<br />
<br />
Create home:<br />
mkdir /home/bb<br />
chown bb.bb /home/bb<br />
cd ~bb<br />
<br />
Copy over and install files:<br />
<pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar .<br />
tar xf bb-linux.tar<br />
cd /home/bb/bbc1.9e-btf/etc</pre><br />
<br />
Configure main bb server:<br />
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts<br />
echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts<br />
<br />
Configure low disk alerts:<br />
<pre>echo "/:90:95<br />
/var:90:95<br />
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab</pre><br />
<br />
vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh<br />
(remove all | SORT xxxx since SORT is broken)<br />
<br />
chmod +r /var/log/messages<br />
<br />
./bbchkcfg.sh <br />
(y to questions)<br />
./bbchkhosts.sh<br />
(ignore ssh errors)<br />
<pre>cd ../..<br />
chown -R bb .<br />
su bb<br />
cd<br />
cd bbc1.9e-btf/src</pre><br />
<br />
<pre>make; make install<br />
cd ..<br />
./runbb.sh start<br />
more BBOUT</pre> <br />
(look for errors)<br />
exit<br />
<br />
<pre>vi /etc/rc.local<br />
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"<br />
</pre><br />
(before the exit 0)<br />
<br />
echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd<br />
<br />
Add f/w rule:<br />
ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984<br />
<br />
<pre>vi ~bb/bbc1.9e-btf/etc/bbdef-client.sh<br />
DFWARN=199<br />
DFPANIC=199</pre><br />
<br />
<br />
=== raid check ===<br />
<br />
==== 3ware ====<br />
<pre><br />
scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli<br />
scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh<br />
scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk<br />
vi /usr/local/sbin/checkraid.sh<br />
:%s/c0/c2/g<br />
<br />
crontab -e<br />
0 0 * * * /usr/local/sbin/3wraidchk</pre><br />
<br />
==== areca ====<br />
<pre><br />
cd /tmp<br />
wget http://www.areca.us/support/s_linux/cli/linuxcli_V1.10.0_120815.zip<br />
unzip linuxcli_V1.10.0_120815.zip<br />
cp linuxcli_V1.10.0_120815/x86_64/cli64 /usr/local/sbin/<br />
chmod 0700 /usr/local/sbin/cli64<br />
cli64 rsf info<br />
</pre><br />
<br />
<pre>scp backup2:/data4/bin/arecaraidchk /usr/local/sbin<br />
scp backup1:/usr/local/sbin/Sendmail.pm /usr/local/sbin<br />
<br />
crontab -e<br />
0 0 * * * /usr/local/sbin/arecaraidchk<br />
</pre><br />
<br />
<pre>cat > /root/verify.sh<br />
cli64 vsf info<br />
cli64 rsf info<br />
cli64 disk info<br />
cli64 event info<br />
echo press enter when ready to run verify ; read x<br />
<br />
cli64 vsf check vol=1<br />
</pre><br />
<br />
=== misc binaries ===<br />
<br />
scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive<br />
vi /usr/local/sbin/snapshot_archive<br />
(remove entries)<br />
<br />
crontab -e<br />
35 4 * * * /usr/local/sbin/snapshot_archive<br />
<br />
scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave<br />
scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone<br />
<br />
Since installing /bin/mail requires all sorts of packages (lame) we write a simple one here...which can only email johncompanies.com addr's unless you add relaying for this host:<br />
<br />
<pre><br />
cat > /bin/mail<br />
#!/usr/bin/perl<br />
use strict;<br />
use warnings;<br />
<br />
use lib '/usr/local/sbin';<br />
use Sendmail qw(sendmail);<br />
<br />
my $sub = $ARGV[1];<br />
my $to = $ARGV[2];<br />
<br />
<br />
my %mail = (<br />
To => $to,<br />
From => $to,<br />
Subject => $sub,<br />
Message => '',<br />
smtp => 'mail.johncompanies.com'<br />
);<br />
sendmail(%mail) || print "Error: $Sendmail::error";<br />
<br />
</pre><br />
<br />
chmod 0700 /bin/mail<br />
<br />
=== mkbackup ===<br />
<br />
mkdir /data/customers<br />
<br />
<pre>cat > /usr/local/sbin/mkbackup<br />
#!/bin/sh<br />
<br />
if test $1; then<br />
cid=$1<br />
else<br />
echo "ERROR: Usage: mkbackup cid GB ip Terminating."<br />
exit<br />
fi<br />
<br />
if test $2; then<br />
gb=$2<br />
else<br />
echo "ERROR: Usage: mkbackup cid GB ip Terminating."<br />
exit<br />
fi<br />
<br />
if test $3; then<br />
ip=$3<br />
else<br />
echo "ERROR: Usage: mkbackup cid GB ip Terminating."<br />
exit<br />
fi<br />
<br />
<br />
if test -e /data/customers/${cid}-file; then<br />
echo "ERROR: /data/customers/${cid}-file exists"<br />
exit<br />
else<br />
echo "touch /data/customers/${cid}-file"<br />
touch /data/customers/${cid}-file<br />
count=`echo $gb|awk '{print $1*1000}'`<br />
echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"<br />
dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count<br />
echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"<br />
/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file<br />
fi<br />
<br />
if test -e /data/customers/$cid; then<br />
echo "ERROR: /data/customers/$cid exists"<br />
exit<br />
else<br />
echo "mkdir /data/customers/${cid}"<br />
mkdir /data/customers/${cid}<br />
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"<br />
mount -o loop /data/customers/${cid}-file /data/customers/$cid<br />
df -h /data/customers/$cid<br />
<br />
echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh<br />
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh<br />
echo "" >> /etc/nfs_backup_mounts.sh<br />
<br />
echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports<br />
/etc/init.d/nfs-kernel-server restart<br />
tail /var/log/messages<br />
fi</pre><br />
<br />
chmod 0700 /usr/local/sbin/mkbackup<br />
<br />
vi /etc/rc.local<br />
add:<br />
/etc/nfs_backup_mounts.sh<br />
<br />
=== samba ===<br />
<br />
apt-get install samba<br />
<br />
vi /etc/samba/smb.conf<br />
<br />
; comment out any mounts, add:<br />
<br />
<pre>[data]<br />
read only = yes<br />
locking = no<br />
path = /data/iso<br />
guest ok = yes</pre><br />
<br />
/etc/init.d/smbd restart<br />
<br />
mkdir /data/iso<br />
<br />
Bring over some stuff from backup2<br />
<br />
<pre>cd /data/iso<br />
scp backup2:/d2/iso/3wfirmware.iso .<br />
scp backup2:/d2/iso/MD5SUMS .<br />
scp backup2:/d2/iso/bootimg.iso .<br />
scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso .<br />
scp backup2:/d2/iso/win98bootcd.iso .<br />
scp backup2:/d2/iso/acronis_bootdisk.iso .<br />
scp backup2:/d2/iso/memtest86-3.2.iso .</pre><br />
<br />
=== Moving from one server to another ===<br />
<br />
Here are the steps you would take to move settings and data from one server to a new backup server:<br />
<br />
* rsync over all /data/customers (we do this cause if we didn't use *-file it would copy over the files AND the data in the mountpoint)<br />
rsync -av --progress --ignore-times *-file root@10.1.2.33:/data/customers/<br />
after umounting all the customers, copy over the (empty) directories separately:<br />
for f in `find . -type d`; do rsync -av $f root@69.55.229.25:/data/customers; done<br />
<br />
* copy mount script<br />
[root@backup3 /data/customers]# scp /etc/nfs_backup_mounts.sh root@69.55.229.25:/etc/nfs_backup_mounts.sh<br />
<br />
* copy rc.local<br />
[root@backup3 /data/customers]# scp /etc/rc.local root@69.55.229.25:/etc/rc.local<br />
<br />
* copy /etc/exports<br />
[root@backup3 /data/customers]# scp /etc/exports root@69.55.229.25:/etc/exports<br />
<br />
* edit /etc/hostname on both machines (set current to oldbackup3)<br />
<br />
* edit /etc/network/interfaces (swap IPs). <br />
<br />
* stop mounts from mounting on old and new servers so it doesnt start with reboot right away:<br />
chmod 000 /etc/nfs_backup_mounts.sh<br />
<br />
* reboot both servers @ same time<br />
<br />
* check everything out<br />
<br />
* run /etc/nfs_backup_mounts.sh on new server<br />
<br />
* if switch port changed update mrtg to reflect correct port pub nic is on (on p20):<br />
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg<br />
<br />
= backup4 =<br />
== Summary ==<br />
This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs. Only FreeBSD virt customers directly accesses this server to perform their backups. <br />
<br />
* Location: castle, cab 3-7<br />
* OS: FreeNAS 9.3 (FreeBSD 9.3)<br />
* Networking: Priv IP: 10.1.2.9/24 AND 10.1.7.9/24, Pub IPs: 69.55.230.6/24 <br />
* Hardware: JC-08014<br />
Intel S5000VSA Motherboard<br />
1 x Intel Xeon E5410 @ 2.33GHz CPU<br />
3ware 9690SA-8I RAID Card w BBU<br />
16GB RAM<br />
Dual power supply.<br />
* Drives: 7 TB (6 x 2TB) ZFS RAIDZ2 array running on JBOD<br />
1 128 GB SSD system drive and 6 drive SATA bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap.<br />
<br />
* GUI management at http://backup4.johncompanies.com<br />
<br />
== Services provided ==<br />
* backup via rsync and nfs<br />
* samba<br />
* nfs<br />
* snmp?<br />
* bigbrother?<br />
<br />
== Usage ==<br />
* all data is stored under /data<br />
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt> ??<br />
* this server serves as an archive for exported db data from bwdb and old flow files. ??<br />
* customers are nfs-moutned under /data/users (/mnt/zfs/users) as zfs ?<br />
<br />
== management scripts ==<br />
* mkbackups?<br />
<br />
mkbackup <cid> GB <ip><br />
<br />
== Cronjobs ==<br />
0 0 * * * /usr/local/sbin/arecaraidchk <br />
RAID checks ?<br />
<br />
35 4 * * * /usr/local/sbin/snapshot_archive<br />
Rotate daily snapshots for infrastructure machine backups<br />
<br />
00 15 * * * /usr/local/sbin/snapshot_rotate<br />
Rotate daily snapshots for customer machine backups<br />
<br />
== Regular maintenance ==<br />
*[[Routine_Maintenance#A|Check on RAID health]]<br />
<br />
== Build ==<br />
<br />
= console = <br />
<br />
== Summary ==<br />
This box's only purpose is to serve as a means to connect to the digi serial multiplexer boxes at castle. Connect to it using the blue (cisco) ribbon cable with the beige RJ-45 to serial connector, 9600 8N1.<br />
<br />
* Location: castle, cab 3-8<br />
* OS: SunOS 5.8 (solaris)<br />
* Networking: Priv IP: 10.1.4.4<br />
* Hardware: Sun Netra<br />
<br />
<br />
To connect to consoles, ssh in as user 'console' and use the <tt>tip</tt> command to connect to devices listed in <tt>/etc/remote</tt><br />
<br />
i.e.<br />
tip switch-p1<br />
tip jail1<br />
<br />
== Configuring digi/ports == <br />
<br />
=== /etc/remote ===<br />
This is where the configuration/mapping for ports and custom names which we use along with the tip command to connect to various ports on the digi switches.<br />
<br />
We have 2 digi's at castle we connect to:<br />
<br />
#3-7 10.1.4.10<br />
virt15:dv=/dev/dty/CO001s:br#38400:el=^C^S^Q^U^D:ie=%$:oe=^D:<br />
virt13:dv=/dev/dty/CO002s:br#115200:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:<br />
<br />
and <br />
<br />
#3-6 10.1.4.11<br />
jail4:dv=/dev/dty/CP001s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:<br />
jail16:dv=/dev/dty/CP002s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:<br />
<br />
The only things you need to edit are the first part (i.e. <tt>jail4</tt>) and the speed (i.e. <tt>9600</tt>). You can decipher which port on the digi each line corresponds to by the <tt>CP001s or CO001s</tt> (port 1 on digi1 and digi2), <tt>CP002s or CO002s</tt> (port 2 on digi1 and digi2)<br />
<br />
=== drpadmin ===<br />
The tool you use to configure a device to a digi box is drpadmin:<br />
<br />
<pre>bash-2.03$ su<br />
Password:<br />
# drpadmin<br />
<br />
Please select an option (a)dd (d)elete (s)how (r)eset (q)uit : s<br />
0 10.1.4.10 32 CO 771 never 1027<br />
1 10.1.4.11 32 CP 771 never 1027<br />
2 65.116.11.2 8 el 771 never 1027<br />
<br />
Please select an option (a)dd (d)elete (s)how (r)eset (q)uit :</pre><br />
<br />
Use those commands above to modify the devices available.<br />
<br />
== Switching IP/hostname ==<br />
<br />
Edit:<br />
/etc/defaultrouter<br />
/etc/hosts<br />
/etc/hostname.hme0<br />
/etc/nodename<br />
Maybe needed to run: # ifconfig hme0 10.1.4.4 up<br />
<br />
= devweb =<br />
<br />
We do web development on devweb.johncompanies.com<br />
<br />
Currently this is a jail running on jail17 / 69.55.230.8<br />
<br />
If the jail is restarted, you will need to manually restart the web service with:<br />
httpsdctl restart<br />
<br />
All website development work should be done here first. It works exactly like and is setup like our [[Management_System_/_Public_Website_/_Signup_/_Account_Manager|main site]].<br />
<br />
<br />
= firewall2 =<br />
<br />
== Summary ==<br />
<br />
This machine is the primary firewall for the entire network at i2b. firewall3 is a hot standby replacement for <br />
firewall2. Both firewall2 and firewall3 should not be connected at the same time since they use the same internal<br />
and external IP addresses.<br />
<br />
* Location: i2b, cab 6<br />
* OS: FreeBSD 6.4 x86<br />
* Networking: Priv IP: 10.1.2.2, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. TODO: describe NIC location/orientation<br />
<br />
* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: 73 GB (2 x 73GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.<br />
<br />
== Services Provided ==<br />
* firewall (ipfw)<br />
* bigbrother for customer machines<br />
<br />
== Firewall Rule Configuration ==<br />
<br />
See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.<br />
<br />
== Disaster Recovery ==<br />
<br />
TODO: need backup f/w and instructions on how to move cables.<br />
<br />
Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)<br />
<br />
TODO<br />
<br />
Here's the config on the live firewall:<br />
<br />
<pre>kern_securelevel_enable="NO"<br />
portmap_enable="NO"<br />
sendmail_enable="NO"<br />
usbd_enable="YES"<br />
gateway_enable="YES"<br />
<br />
xntpd_enable="YES"<br />
nfs_client_enable="YES"<br />
nfs_reserved_port_only="YES"<br />
inetd_enable="YES"<br />
inetd_flags="-wW -a 10.1.2.2"<br />
<br />
fsck_y_enable="YES"<br />
background_fsck="NO"<br />
<br />
defaultrouter="66.181.18.2"<br />
hostname="firewall2.johncompanies.com"<br />
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"<br />
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"<br />
ifconfig_bge1_alias0="inet 69.55.231.1 netmask 255.255.255.0"<br />
ifconfig_bge1_alias1="inet 65.50.228.1 netmask 255.255.255.0"<br />
ifconfig_bge1_alias2="inet 65.50.229.1 netmask 255.255.255.0"<br />
ifconfig_bge1_alias3="inet 65.50.230.1 netmask 255.255.255.0"<br />
ifconfig_bge1_alias4="inet 65.50.231.1 netmask 255.255.255.0"<br />
ifconfig_bge1_alias5="inet 65.50.232.1 netmask 255.255.255.0"<br />
ifconfig_bge1_alias6="inet 65.50.233.1 netmask 255.255.255.0"<br />
ifconfig_bge1_alias7="inet 65.50.234.1 netmask 255.255.255.0"<br />
ifconfig_bge1_alias8="inet 65.50.235.1 netmask 255.255.255.0"<br />
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"<br />
sshd_enable="YES"<br />
usbd_enable="YES"<br />
</pre><br />
<br />
<br />
== Cronjobs ==<br />
30 3 * * * /usr/local/etc/rsync.backup<br />
Backup to backup3<br />
<br />
0 0 1 * * /sbin/ipfw zero<br />
0 0 1 * * /sbin/ipfw del 3 4 5<br />
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap). <br />
<br />
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count<br />
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count<br />
Capture counts periodically<br />
<br />
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;<br />
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.<br />
<br />
*/5 * * * * /usr/local/sbin/lsiraidchk<br />
Checking the health of the RAID array<br />
<br />
<br />
== DOS attacks ==<br />
<br />
See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.<br />
<br />
<br />
== build ==<br />
<br />
<pre>partition map:<br />
/ 58g<br />
swap 4g<br />
/var 512m<br />
/tmp 512m<br />
/usr 5.5g<br />
<br />
4. edit /etc/make.conf <br />
echo "WITHOUT_X11=yes \<br />
KERNCONF=firewall2 \<br />
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf<br />
<br />
5. add settings to /boot/loader.conf and /boot.config<br />
<br />
echo "-Dh" >> /boot.config<br />
<br />
echo 'console="comconsole,vidconsole" \<br />
boot_multicons="YES" \<br />
boot_serial="YES" \<br />
comconsole_speed="115200"' >> /boot/loader.conf<br />
<br />
<br />
6. turn off all ttyv's except 0 and 1 in /etc/ttys<br />
also turn on ttyd0, change type to vt100:<br />
vi /etc/ttys<br />
<br />
ttyv2 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv3 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv4 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv5 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv6 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv7 "/usr/libexec/getty Pc" cons25 off secure<br />
# Serial terminals<br />
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.<br />
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure<br />
<br />
kill -1 1<br />
<br />
on console server:<br />
vi /etc/remote<br />
(rename port to jail8 depending on where and which digi plugged into)<br />
test serial console<br />
<br />
<br />
7. populate hosts<br />
echo "69.55.230.10 backup2" >> /etc/hosts<br />
echo "69.55.230.11 backup1" >> /etc/hosts<br />
echo "10.1.2.3 backup3" >> /etc/hosts<br />
<br />
8. put key in authorized_keys on backup3<br />
cd<br />
ssh-keygen -t dsa -b 1024 <br />
(default location, leave password blank)<br />
<br />
Punch a hole in firewall1 to allow traffic to backup servers @ castle:<br />
<br />
ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22<br />
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22<br />
<br />
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys' <br />
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' <br />
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys' <br />
<br />
confirm that you can ssh to backup3 and backup 2 without getting a login prompt<br />
<br />
ssh backup3 hostname<br />
<br />
ssh backup2 hostname<br />
<br />
ssh backup1 hostname<br />
<br />
<br />
10. edit root's path and login script:<br />
vi /root/.cshrc<br />
<br />
Change alias entries (add G):<br />
alias la ls -aG<br />
alias lf ls -FAG<br />
alias ll ls -lAG<br />
alias ls ls -AG<br />
alias mbm mb mount<br />
alias mbu mb umount<br />
<br />
and alter the prompt, set the following:<br />
set prompt = "`/bin/hostname -s` %/# "<br />
<br />
11. install cvsup<br />
cd /usr/ports/net/cvsup-without-gui <br />
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null<br />
<br />
12. get latest sources for this release:<br />
cd /usr/src <br />
echo "*default host=cvsup4.freebsd.org\<br />
*default base=/usr\<br />
*default prefix=/usr\<br />
*default release=cvs tag=RELENG_6_4\<br />
*default delete use-rel-suffix\<br />
*default compress\<br />
src-all" > sup<br />
<br />
cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null<br />
<br />
13. configure new kernel. <br />
<br />
cd /usr/src/sys/i386/conf <br />
scp backup2:/mnt/data4/build/freebsd/firewall2-6.4 ./firewall2<br />
<br />
15. build, install kernel and world<br />
<br />
cd /boot<br />
<br />
mv kernel kernel.GENERIC<br />
cd kernel.GENERIC<br />
cd /usr/src<br />
make buildkernel installkernel<br />
<br />
make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null<br />
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)<br />
make installworld <br />
(2450: 3min, supermicro: 1min, 2950: :34)<br />
mergemaster -i<br />
<br />
17. populate /etc/rc.conf with IPs and NFS settings<br />
vi /etc/rc.conf<br />
<br />
kern_securelevel_enable="NO"<br />
portmap_enable="NO"<br />
sendmail_enable="NO"<br />
usbd_enable="YES"<br />
gateway_enable="YES"<br />
<br />
xntpd_enable="YES"<br />
nfs_client_enable="YES"<br />
nfs_reserved_port_only="YES"<br />
inetd_enable="YES"<br />
inetd_flags="-wW -a 10.1.2.2"<br />
<br />
ifconfig_bce1="inet 10.1.2.2 netmask 255.255.255.0"<br />
fsck_y_enable="YES"<br />
background_fsck="NO"<br />
<br />
defaultrouter="66.181.18.2"<br />
hostname="firewall2.johncompanies.com"<br />
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"<br />
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"<br />
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"<br />
sshd_enable="YES"<br />
usbd_enable="YES"<br />
<br />
20. reboot. Confirm new kernel is loaded<br />
<br />
uname -a<br />
<br />
21. update ports:<br />
cd /usr/ports<br />
echo "*default host=cvsup4.FreeBSD.org\<br />
*default base=/usr\<br />
*default prefix=/usr\<br />
*default release=cvs tag=RELENG_6_4\<br />
*default delete use-rel-suffix\<br />
*default compress\<br />
ports-all tag=." > sup<br />
<br />
cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null<br />
<br />
22. Install raid mgmt tool<br />
<br />
# linux base<br />
cd /usr/ports/devel/libtool22<br />
make install base<br />
<br />
cd /usr/ports/emulators/linux_base-fc4<br />
make install clean<br />
<br />
#linux-megamgr-5.20<br />
cd /usr/ports/sysutils/linux-megamgr<br />
make install clean<br />
<br />
# megarc-1.51<br />
cd /usr/ports/sysutils/megarc<br />
make install clean<br />
<br />
Test:<br />
rehash; megarc -ldInfo -a0 -l0<br />
<br />
23. install rsync from ports<br />
cd /usr/ports/net/rsync<br />
make install clean<br />
<br />
choose default options<br />
<br />
25. install bb client<br />
adduser<br />
Username: bb<br />
Full name: bb<br />
Uid (Leave empty for default): 1984<br />
Login group [bb]:<br />
Login group is bb. Invite bb into other groups? []:<br />
Login class [default]:<br />
Shell (sh csh tcsh nologin) [sh]: <br />
Home directory [/home/bb]:<br />
Use password-based authentication? [yes]:<br />
Use an empty password? (yes/no) [no]:<br />
Use a random password? (yes/no) [no]: yes<br />
Lock out the account after creation? [no]:<br />
Username : bb<br />
Password : <random><br />
Full Name : bb<br />
Uid : 1984<br />
Class :<br />
Groups : bb<br />
Home : /home/bb<br />
Shell : /bin/sh<br />
Locked : no<br />
OK? (yes/no): yes<br />
<br />
cd /usr/home/bb<br />
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .<br />
tar xvf bb-freebsd.tar<br />
<br />
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:<br />
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \<br />
10.1.2.1 firewall2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts<br />
<br />
vi /home/bb/bbc1.9e-btf/ext/openfiles <br />
MACHINE="firewall2,johncompanies,com" # HAS TO BE IN A,B,C FORM<br />
<br />
cd /usr/home/bb/bbc1.9e-btf/etc<br />
./bbchkcfg.sh <br />
(y to questions)<br />
./bbchkhosts.sh<br />
(ignore ssh errors)<br />
cd ../..<br />
chown -R bb .<br />
su bb<br />
cd<br />
cd bbc1.9e-btf/src<br />
make; make install<br />
cd ..<br />
<br />
vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh<br />
$1 $TOPARGS > $BBTMP/TOP.$$<br />
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$<br />
<br />
./runbb.sh start<br />
more BBOUT <br />
(look for errors)<br />
exit<br />
<br />
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh<br />
chmod +x /usr/local/etc/rc.d/bb.sh<br />
<br />
Punch a hole in the firewall to allow it to communicate with bb monitor:<br />
<br />
ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2<br />
<br />
<br />
27. configure bb on mail:<br />
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts<br />
66.181.18.3 firewall2.johncompanies.com # ssh<br />
<br />
su bb<br />
cd<br />
bbsrc/bb/runbb.sh restart ; exit<br />
<br />
29. configure ntp<br />
echo "server 10.1.2.1" > /etc/ntp.conf<br />
<br />
/usr/sbin/ntpd -p /var/run/ntpd.pid<br />
sleep 2; ntpq -p<br />
(confirm it’s able to reach our time server)<br />
<br />
echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh<br />
chmod 0700 /usr/local/etc/rc.d/ntp.sh<br />
<br />
30. fwd and reverse lookups on ns1c<br />
vr johncompanies.com<br />
(edit the PTR too)<br />
<br />
<br />
33. setup backups<br />
echo '#\!/bin/sh\<br />
backupdir=/data/firewall2/current\<br />
\<br />
## ENTRY /etc ' > /usr/local/etc/backup.config<br />
<br />
on backup3:<br />
setup backup dirs:<br />
ssh backup3 mkdir -p /data/firewall2/current<br />
<br />
on backup3, add the system to <br />
vi /usr/local/sbin/snapshot_archive<br />
<br />
scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup <br />
vi /usr/local/etc/rsync.backup <br />
backup1 > backup3<br />
<br />
crontab -e<br />
1 0 * * * /usr/local/etc/rsync.backup<br />
<br />
<br />
34. mkdir /root/logs<br />
<br />
35. edit sshd_config for security<br />
vi /etc/ssh/sshd_config<br />
ListenAddress 66.181.18.3<br />
ListenAddress 10.1.2.1<br />
<br />
kill -1 `cat /var/run/sshd.pid`<br />
<br />
35. raid chk<br />
<br />
cat > /usr/local/sbin/lsiraidchk<br />
#!/usr/bin/perl<br />
<br />
my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;<br />
<br />
foreach (@out) {<br />
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }<br />
#print $_;<br />
}<br />
<br />
36. add crontab entries<br />
crontab -e<br />
30 3 * * * /usr/local/etc/rsync.backup<br />
0 0 1 * * /sbin/ipfw zero<br />
0 0 1 * * /sbin/ipfw del 3 4 5<br />
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count<br />
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count<br />
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl; <br />
*/5 * * * * /usr/local/sbin/lsiraidchk<br />
<br />
#10 0 * * * rm /var/spool/clientmqueue/*<br />
<br />
scp /etc/makefwrules.pl user@64.163.14.48:~<br />
scp /etc/makepiperules.pl user@64.163.14.48:~<br />
mv /home/user/makefwrules.pl /etc<br />
mv /home/user/makepiperules.pl /etc<br />
touch /etc/firewall.sh<br />
mkdir /etc/oldrules/<br />
<br />
other binaries<br />
<br />
scp /usr/local/bin/rulemaker user@64.163.14.48:~<br />
mv ~user/rulemaker /usr/local/sbin<br />
scp ~user/Sendmail.pm user@64.163.14.48:~<br />
scp ~user/doswatch.pl user@64.163.14.48:~<br />
<br />
Setup basic ruleset<br />
<br />
ipfw add 00009 count udp from any to any<br />
ipfw add 00010 allow tcp from any to any established<br />
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss<br />
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18<br />
ipfw add 00012 deny tcp from any to any tcpflags syn,fin<br />
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg<br />
ipfw add 00012 allow icmp from any to any<br />
ipfw add 00014 deny tcp from any to any dst-port 135<br />
ipfw add 00150 skipto 65535 ip from any to any via em1 in<br />
<br />
IPKVM3:<br />
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139<br />
00098 deny ip from any to 69.55.230.10 dst-port 139<br />
</pre><br />
<br />
<br />
= firewall3 =<br />
<br />
== Summary ==<br />
<br />
This machine is the backup firewall for the network at i2b. <br />
<br />
* Location: i2b, cab ?<br />
* OS: FreeBSD 9.1 amd64<br />
* Networking: Priv IP: 10.1.2.5, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. <br />
<br />
<pre><br />
<br />
The internal network NIC is the left one on the motherboard (69.55.229.1/24, ...).<br />
The external network NIC is the right one on the motherboard (66.181.18.3/28).<br />
The PCI ethernet card is connected to our private network (10.1.2.5/24).<br />
<br />
</pre><br />
<br />
* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: 160 GB (2 x 160GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.<br />
<br />
== Services Provided ==<br />
* firewall (ipfw)<br />
* bigbrother<br />
<br />
== Firewall Rule Configuration ==<br />
<br />
See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.<br />
<br />
== Disaster Recovery ==<br />
<br />
'''To put the backup firewall3 into service:'''<br />
<br />
<pre><br />
Move the internal cable (to our networks) from firewall2 to em1 which is the left most ethernet port (69.55.229.1).<br />
Move the external cable (to outside world) from firewall2 to em0 which is the port to the right on the motherboard (66.181.18.3).<br />
The PCI ethernet port (fxp0) should already be connected to private network (10.1.2.5).<br />
</pre><br />
<br />
Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)<br />
<br />
<pre><br />
kern_securelevel_enable="NO"<br />
portmap_enable="NO"<br />
sendmail_enable="NO"<br />
usbd_enable="YES"<br />
gateway_enable="YES"<br />
<br />
xntpd_enable="YES"<br />
nfs_client_enable="YES"<br />
nfs_reserved_port_only="YES"<br />
inetd_enable="YES"<br />
inetd_flags="-wW -a 10.1.2.5"<br />
<br />
fsck_y_enable="YES"<br />
background_fsck="NO"<br />
<br />
defaultrouter="66.181.18.2"<br />
hostname="firewall3.johncompanies.com"<br />
ifconfig_em0="inet 66.181.18.3 netmask 255.255.255.224"<br />
<br />
ifconfig_em1="inet 69.55.229.1 netmask 255.255.255.0"<br />
ifconfig_em1_alias0="inet 69.55.231.1 netmask 255.255.255.0"<br />
<br />
# ifconfig_em1_alias1="inet 65.50.228.1 netmask 255.255.255.0"<br />
# ifconfig_em1_alias2="inet 65.50.229.1 netmask 255.255.255.0"<br />
# ifconfig_em1_alias3="inet 65.50.230.1 netmask 255.255.255.0"<br />
# ifconfig_em1_alias4="inet 65.50.231.1 netmask 255.255.255.0"<br />
# ifconfig_em1_alias5="inet 65.50.232.1 netmask 255.255.255.0"<br />
# ifconfig_em1_alias6="inet 65.50.233.1 netmask 255.255.255.0"<br />
# ifconfig_em1_alias7="inet 65.50.234.1 netmask 255.255.255.0"<br />
# ifconfig_em1_alias8="inet 65.50.235.1 netmask 255.255.255.0"<br />
<br />
ifconfig_fxp0="inet 10.1.2.5 netmask 255.255.255.0"<br />
<br />
sshd_enable="YES"<br />
usbd_enable="YES"<br />
</pre><br />
<br />
== Cronjobs ==<br />
30 3 * * * /usr/local/etc/rsync.backup<br />
Backup to backup3<br />
<br />
0 0 1 * * /sbin/ipfw zero<br />
0 0 1 * * /sbin/ipfw del 3 4 5<br />
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap). <br />
<br />
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count<br />
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count<br />
Capture counts periodically<br />
<br />
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;<br />
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.<br />
<br />
*/5 * * * * /usr/local/sbin/lsiraidchk<br />
Checking the health of the RAID array<br />
<br />
<br />
== DOS attacks ==<br />
<br />
See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.<br />
<br />
<br />
== build ==<br />
<br />
<pre>partition map:<br />
/ 58g<br />
swap 4g<br />
/var 512m<br />
/tmp 512m<br />
/usr 5.5g<br />
<br />
4. edit /etc/make.conf <br />
echo "WITHOUT_X11=yes \<br />
KERNCONF=firewall3 \<br />
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf<br />
<br />
5. add settings to /boot/loader.conf and /boot.config<br />
<br />
echo "-Dh" >> /boot.config<br />
<br />
echo 'console="comconsole,vidconsole" \<br />
boot_multicons="YES" \<br />
boot_serial="YES" \<br />
comconsole_speed="115200"' >> /boot/loader.conf<br />
<br />
<br />
6. turn off all ttyv's except 0 and 1 in /etc/ttys<br />
also turn on ttyd0, change type to vt100:<br />
vi /etc/ttys<br />
<br />
ttyv2 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv3 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv4 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv5 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv6 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv7 "/usr/libexec/getty Pc" cons25 off secure<br />
# Serial terminals<br />
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.<br />
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure<br />
<br />
kill -1 1<br />
<br />
on console server:<br />
vi /etc/remote<br />
(rename port to jail8 depending on where and which digi plugged into)<br />
test serial console<br />
<br />
<br />
7. populate hosts<br />
echo "69.55.230.10 backup2" >> /etc/hosts<br />
echo "69.55.230.11 backup1" >> /etc/hosts<br />
echo "10.1.2.3 backup3" >> /etc/hosts<br />
<br />
8. put key in authorized_keys on backup3<br />
cd<br />
ssh-keygen -t dsa -b 1024 <br />
(default location, leave password blank)<br />
<br />
Punch a hole in firewall1 to allow traffic to backup servers @ castle:<br />
<br />
ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22<br />
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22<br />
<br />
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys' <br />
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' <br />
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys' <br />
<br />
confirm that you can ssh to backup3 and backup 2 without getting a login prompt<br />
<br />
ssh backup3 hostname<br />
<br />
ssh backup2 hostname<br />
<br />
ssh backup1 hostname<br />
<br />
<br />
10. edit root's path and login script:<br />
vi /root/.cshrc<br />
<br />
Change alias entries (add G):<br />
alias la ls -aG<br />
alias lf ls -FAG<br />
alias ll ls -lAG<br />
alias ls ls -AG<br />
alias mbm mb mount<br />
alias mbu mb umount<br />
<br />
and alter the prompt, set the following:<br />
set prompt = "`/bin/hostname -s` %/# "<br />
<br />
11. install cvsup<br />
cd /usr/ports/net/cvsup-without-gui <br />
make install clean; rehash; mail -s 'cvs installed' 8583619553@vtext.com < /dev/null<br />
<br />
12. get latest sources for this release:<br />
cd /usr/src <br />
echo "*default host=cvsup4.freebsd.org\<br />
*default base=/usr\<br />
*default prefix=/usr\<br />
*default release=cvs tag=RELENG_9_1\<br />
*default delete use-rel-suffix\<br />
*default compress\<br />
src-all" > sup<br />
<br />
cvsup sup ; mail -s 'cvs sup done' 8583619553@vtext.com < /dev/null<br />
<br />
13. configure new kernel. <br />
<br />
cd /usr/src/sys/amd64/conf <br />
scp backup2:/mnt/data4/build/freebsd/firewall3-9.1 ./firewall3<br />
<br />
15. build, install kernel and world<br />
<br />
cd /boot<br />
<br />
mv kernel kernel.GENERIC<br />
cd kernel.GENERIC<br />
cd /usr/src<br />
make buildkernel installkernel<br />
<br />
make buildworld ; mail -s 'buildworld done' 8583619553@vtext.com < /dev/null<br />
(supermicro: 2:15 mins, 2950: 38? mins)<br />
make installworld <br />
(2450: 3min, supermicro: 1min, 2950: :34)<br />
mergemaster -i<br />
<br />
17. populate /etc/rc.conf with IPs and NFS settings<br />
vi /etc/rc.conf<br />
<br />
kern_securelevel_enable="NO"<br />
portmap_enable="NO"<br />
sendmail_enable="NO"<br />
usbd_enable="YES"<br />
gateway_enable="YES"<br />
<br />
xntpd_enable="YES"<br />
nfs_client_enable="YES"<br />
nfs_reserved_port_only="YES"<br />
<br />
fsck_y_enable="YES"<br />
background_fsck="NO"<br />
<br />
hostname="firewall3.johncompanies.com"<br />
# external network<br />
ifconfig_em0="inet 66.181.18.3 netmask 255.255.255.224"<br />
<br />
# internal network<br />
ifconfig_em1="inet 69.55.229.1 netmask 255.255.255.0"<br />
ifconfig_em1_alias0="inet 69.55.231.1 netmask 255.255.255.0"<br />
<br />
ifconfig_em1_alias1="inet 65.50.228.1 netmask 255.255.255.0"<br />
ifconfig_em1_alias2="inet 65.50.229.1 netmask 255.255.255.0"<br />
ifconfig_em1_alias3="inet 65.50.230.1 netmask 255.255.255.0"<br />
ifconfig_em1_alias4="inet 65.50.231.1 netmask 255.255.255.0"<br />
ifconfig_em1_alias5="inet 65.50.232.1 netmask 255.255.255.0"<br />
ifconfig_em1_alias6="inet 65.50.233.1 netmask 255.255.255.0"<br />
ifconfig_em1_alias7="inet 65.50.234.1 netmask 255.255.255.0"<br />
ifconfig_em1_alias8="inet 65.50.235.1 netmask 255.255.255.0"<br />
<br />
defaultrouter="66.181.18.2"<br />
<br />
# private network<br />
ifconfig_fxp0="inet 10.1.2.5 netmask 255.255.255.0"<br />
<br />
inetd_enable="YES"<br />
inetd_flags="-wW -a 10.1.2.5"<br />
<br />
sshd_enable="YES"<br />
usbd_enable="YES"<br />
ntpd_enable="YES"<br />
# powerd_enable="YES"<br />
<br />
<br />
20. reboot. Confirm new kernel is loaded<br />
<br />
uname -a<br />
<br />
21. update ports:<br />
cd /usr/ports<br />
<br />
echo "*default host=cvsup4.FreeBSD.org\<br />
*default base=/usr\<br />
*default prefix=/usr\<br />
*default release=cvs tag=RELENG_9_1\<br />
*default delete use-rel-suffix\<br />
*default compress\<br />
ports-all tag=." > sup<br />
<br />
cvsup sup; mail -s 'cvs sup ports done' 8583619553@vtext.com < /dev/null<br />
<br />
22. Install raid mgmt tool<br />
<br />
# linux base<br />
cd /usr/ports/devel/libtool22<br />
make install base<br />
<br />
cd /usr/ports/emulators/linux_base-fc4<br />
make install clean<br />
<br />
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin<br />
cd /usr/local/sbin <br />
tar xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz<br />
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz<br />
<br />
23. install rsync from ports<br />
cd /usr/ports/net/rsync<br />
make install clean<br />
<br />
choose default options<br />
<br />
25. install bb client<br />
adduser<br />
Username: bb<br />
Full name: bb<br />
Uid (Leave empty for default): 1984<br />
Login group [bb]:<br />
Login group is bb. Invite bb into other groups? []:<br />
Login class [default]:<br />
Shell (sh csh tcsh nologin) [sh]: <br />
Home directory [/home/bb]:<br />
Use password-based authentication? [yes]:<br />
Use an empty password? (yes/no) [no]:<br />
Use a random password? (yes/no) [no]: yes<br />
Lock out the account after creation? [no]:<br />
Username : bb<br />
Password : <random><br />
Full Name : bb<br />
Uid : 1984<br />
Class :<br />
Groups : bb<br />
Home : /home/bb<br />
Shell : /bin/sh<br />
Locked : no<br />
OK? (yes/no): yes<br />
<br />
cd /usr/home/bb<br />
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .<br />
tar xvf bb-freebsd.tar<br />
<br />
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:<br />
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \<br />
10.1.2.5 firewall3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts<br />
<br />
vi /home/bb/bbc1.9e-btf/ext/openfiles <br />
MACHINE="firewall3,johncompanies,com" # HAS TO BE IN A,B,C FORM<br />
<br />
cd /usr/home/bb/bbc1.9e-btf/etc<br />
./bbchkcfg.sh <br />
(y to questions)<br />
./bbchkhosts.sh<br />
(ignore ssh errors)<br />
cd ../..<br />
chown -R bb .<br />
su bb<br />
cd<br />
cd bbc1.9e-btf/src<br />
make; make install<br />
cd ..<br />
<br />
vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh<br />
$1 $TOPARGS > $BBTMP/TOP.$$<br />
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$<br />
<br />
./runbb.sh start<br />
more BBOUT <br />
(look for errors)<br />
exit<br />
<br />
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh<br />
chmod +x /usr/local/etc/rc.d/bb.sh<br />
<br />
Punch a hole in the firewall to allow it to communicate with bb monitor:<br />
<br />
ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2<br />
<br />
<br />
27. configure bb on mail:<br />
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts<br />
66.181.18.3 firewall3.johncompanies.com # ssh<br />
<br />
su bb<br />
cd<br />
bbsrc/bb/runbb.sh restart ; exit<br />
<br />
29. configure ntp<br />
echo "server 10.1.2.1" > /etc/ntp.conf<br />
<br />
/usr/sbin/ntpd -p /var/run/ntpd.pid<br />
sleep 2; ntpq -p<br />
(confirm it’s able to reach our time server)<br />
<br />
echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh<br />
chmod 0700 /usr/local/etc/rc.d/ntp.sh<br />
<br />
30. fwd and reverse lookups on ns1c<br />
vr johncompanies.com<br />
(edit the PTR too)<br />
<br />
<br />
33. setup backups<br />
echo '#\!/bin/sh\<br />
backupdir=/data/firewall2/current\<br />
\<br />
## ENTRY /etc ' > /usr/local/etc/backup.config<br />
<br />
on backup3:<br />
setup backup dirs:<br />
ssh backup3 mkdir -p /data/firewall2/current<br />
<br />
on backup3, add the system to <br />
vi /usr/local/sbin/snapshot_archive<br />
<br />
scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup <br />
vi /usr/local/etc/rsync.backup <br />
backup1 > backup3<br />
<br />
crontab -e<br />
1 0 * * * /usr/local/etc/rsync.backup<br />
<br />
<br />
34. mkdir /root/logs<br />
<br />
35. edit sshd_config for security<br />
vi /etc/ssh/sshd_config<br />
ListenAddress 66.181.18.3<br />
ListenAddress 10.1.2.5<br />
<br />
kill -1 `cat /var/run/sshd.pid`<br />
<br />
35. raid chk<br />
<br />
cat > /usr/local/sbin/lsiraidchk<br />
#!/usr/bin/perl<br />
<br />
my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;<br />
<br />
foreach (@out) {<br />
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }<br />
#print $_;<br />
}<br />
<br />
36. add crontab entries<br />
crontab -e<br />
30 3 * * * /usr/local/etc/rsync.backup<br />
0 0 1 * * /sbin/ipfw zero<br />
0 0 1 * * /sbin/ipfw del 3 4 5<br />
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count<br />
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count<br />
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl; <br />
*/5 * * * * /usr/local/sbin/lsiraidchk<br />
<br />
#10 0 * * * rm /var/spool/clientmqueue/*<br />
<br />
scp /etc/makefwrules.pl user@64.163.14.48:~<br />
scp /etc/makepiperules.pl user@64.163.14.48:~<br />
mv /home/user/makefwrules.pl /etc<br />
mv /home/user/makepiperules.pl /etc<br />
touch /etc/firewall.sh<br />
mkdir /etc/oldrules/<br />
<br />
other binaries<br />
<br />
scp /usr/local/bin/rulemaker user@64.163.14.48:~<br />
mv ~user/rulemaker /usr/local/sbin<br />
scp ~user/Sendmail.pm user@64.163.14.48:~<br />
scp ~user/doswatch.pl user@64.163.14.48:~<br />
<br />
Setup basic ruleset<br />
<br />
ipfw add 00009 count udp from any to any<br />
ipfw add 00010 allow tcp from any to any established<br />
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss<br />
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18<br />
ipfw add 00012 deny tcp from any to any tcpflags syn,fin<br />
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg<br />
ipfw add 00012 allow icmp from any to any<br />
ipfw add 00014 deny tcp from any to any dst-port 135<br />
ipfw add 00150 skipto 65535 ip from any to any via em1 in<br />
<br />
IPKVM3:<br />
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139<br />
00098 deny ip from any to 69.55.230.10 dst-port 139<br />
</pre><br />
<br />
= wiki =<br />
<br />
The wiki (mediawiki) runs on nat2 in a jail running off 69.55.229.8<br />
<br />
The backup wiki lives on virt13 in CT 5 / 69.55.230.18<br />
<br />
== Setup jail ==<br />
<br />
<pre><br />
mkdir /mnt/data1/wiki-dir<br />
cd /usr/src<br />
make installworld DESTDIR=/mnt/data1/wiki-dir<br />
cd etc<br />
make distribution DESTDIR=/mnt/data1/wiki-dir<br />
<br />
mount -t devfs devfs /mnt/data1/wiki-dir/dev<br />
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset <br />
<br />
cd /mnt/data1/wiki-dir<br />
<br />
ln -sf dev/null kernel<br />
<br />
scp jail9:/usr/local/sbin/jkill /mnt/data1/wiki-dir/sbin<br />
<br />
jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh<br />
csh<br />
<br />
touch /etc/fstab<br />
echo 'network_interfaces=""\<br />
hostname="wiki.johncompanies.com"\<br />
kern_securelevel_enable="NO"\<br />
sendmail_enable="YES"\<br />
sshd_enable="YES"' > /etc/rc.conf<br />
<br />
echo "nameserver 69.55.229.3\<br />
nameserver 69.55.225.225" >> /etc/resolv.conf <br />
<br />
vi /etc/crontab<br />
<br />
(remove the adjkerntz lines )<br />
<br />
vi /etc/periodic/security/100.chksetuid<br />
<br />
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`<br />
with: MP='/' (use single quotes) <br />
<br />
mkdir -p /usr/compat/linux/dev<br />
<br />
adduser<br />
<br />
Username: user<br />
Full name: user<br />
Uid (Leave empty for default):<br />
Login group [user]:<br />
Login group is user. Invite user into other groups? []: wheel<br />
Login class [default]:<br />
Shell (sh csh tcsh nologin) [sh]:<br />
Home directory [/home/user]:<br />
Home directory permissions (Leave empty for default):<br />
Use password-based authentication? [yes]:<br />
Use an empty password? (yes/no) [no]:<br />
Use a random password? (yes/no) [no]: y<br />
Lock out the account after creation? [no]:<br />
Username : user<br />
Password : <random><br />
Full Name : user<br />
Uid : 1001<br />
Class :<br />
Groups : user<br />
Home : /home/user<br />
Home Mode :<br />
Shell : /bin/sh<br />
Locked : no<br />
OK? (yes/no): y<br />
adduser: INFO: Successfully added (user) to the user database.<br />
adduser: INFO: Password for (user) is: 901gmYjO<br />
Add another user? (yes/no): n<br />
Goodbye! <br />
<br />
vi /usr/home/user/.profile <br />
TERM=vt100; export TERM<br />
<br />
tzsetup<br />
<br />
newaliases <br />
<br />
rm /sbin/halt /sbin/reboot<br />
ln /sbin/jkill /sbin/halt<br />
ln /sbin/jkill /sbin/reboot<br />
<br />
vi /etc/syslog.conf<br />
#*.err;kern.warning;auth.notice;mail.crit /dev/console<br />
*.err;kern.warning;auth.notice;mail.crit /var/log/messages <br />
<br />
exit<br />
exit<br />
<br />
cd libexec<br />
chflags noschg ld-elf32.so.1<br />
chflags noschg ld-elf.so.1<br />
mv ld-elf32.so.1 ld-elf32.so.1-orig<br />
ln ld-elf.so.1 ld-elf32.so.1<br />
chflags schg ld-elf.so.1<br />
chflags schg ld-elf32.so.1<br />
<br />
cp -r /usr/ports /mnt/data1/wiki-dir/usr <br />
<br />
cat > /usr/local/etc/rc.d/wiki.sh<br />
mount -t devfs devfs /mnt/data1/wiki-dir/dev/<br />
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset<br />
jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh /etc/rc<br />
<br />
chmod 0700 /usr/local/etc/rc.d/wiki.sh<br />
</pre><br />
<br />
== mediawiki setup ==<br />
<br />
<pre><br />
<br />
cd /usr/ports/net/rsync<br />
make install clean<br />
<br />
cd /usr/ports/distfiles/<br />
fetch http://downloads.mysql.com/archives/mysql-5.5/mysql-5.5.4-m3.tar.gz<br />
cd /usr/ports/databases/mysql55-server<br />
make install clean<br />
<br />
cd /usr/ports/distfiles/<br />
fetch http://downloads.php.net/johannes/php-5.3.2.tar.bz2<br />
cd /usr/ports/lang/php52<br />
make install clean<br />
(build apache module)<br />
<br />
cd /usr/ports/lang/php5-extensions<br />
make install clean<br />
<br />
cd /usr/ports/www/apache22<br />
make install clean<br />
<br />
cd /usr/local/www/<br />
fetch http://download.wikimedia.org/mediawiki/1.19/mediawiki-1.19.2.tar.gz<br />
tar xzf mediawiki-1.19.2.tar.gz<br />
mv mediawiki-1.19.2 wiki<br />
<br />
vi /usr/local/etc/apache22/httpd.conf<br />
DocumentRoot "/usr/local/www/"<br />
<br />
Include etc/apache22/extra/vhost-wiki.conf<br />
Listen 443<br />
<br />
<IfModule mod_php5.c><br />
<FilesMatch "\.ph(p3?|tml)$"><br />
SetHandler application/x-httpd-php<br />
</FilesMatch><br />
<FilesMatch "\.phps$"><br />
SetHandler application/x-httpd-php-source<br />
</FilesMatch><br />
# To re-enable php in user directories comment the following lines<br />
# (from <IfModule ...> to </IfModule>.) Do NOT set it to On as it<br />
# prevents .htaccess files from disabling it.<br />
<IfModule mod_userdir.c><br />
<Directory /home/*/public_html><br />
php_admin_value engine Off<br />
</Directory><br />
</IfModule><br />
</IfModule><br />
<br />
<br />
cat > /usr/local/etc/apache22/extra/vhost-wiki.conf <br />
<VirtualHost *:443><br />
ServerAdmin support@johncompanies.com<br />
<br />
DocumentRoot /usr/local/www/wiki<br />
# <Directory /><br />
# Options FollowSymLinks<br />
# AllowOverride None<br />
# Order deny,allow<br />
# </Directory><br />
<Directory /usr/local/www/wiki><br />
Options Indexes FollowSymLinks MultiViews<br />
Deny from all<br />
AllowOverride AuthConfig<br />
Order allow,deny<br />
DirectoryIndex index.php<br />
#Allow from 69.55.233.195<br />
#Allow from boody.dyndns.org<br />
</Directory><br />
<br />
ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/<br />
<Directory "/usr/lib/cgi-bin"><br />
AllowOverride None<br />
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch<br />
Order allow,deny<br />
Allow from all<br />
</Directory><br />
<br />
ErrorLog /var/log/httpd-error.log<br />
<br />
# Possible values include: debug, info, notice, warn, error, crit,<br />
# alert, emerg.<br />
LogLevel warn<br />
<br />
CustomLog /var/log/httpd-access.log combined<br />
<br />
Alias /doc/ "/usr/share/doc/"<br />
<Directory "/usr/share/doc/"><br />
Options Indexes MultiViews FollowSymLinks<br />
AllowOverride None<br />
Order deny,allow<br />
Deny from all<br />
Allow from 127.0.0.0/255.0.0.0 ::1/128<br />
</Directory><br />
<br />
SSLEngine on<br />
SSLCertificateFile /usr/local/etc/apache22/ssl/server.crt<br />
SSLCertificateKeyFile /usr/local/etc/apache22/ssl/server.key<br />
<br />
</VirtualHost><br />
<br />
mkdir ssl<br />
cd ssl<br />
<br />
openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key<br />
US<br />
CA<br />
San Diego<br />
johncompanies.com<br />
johncompanies.com<br />
wiki.johncompanies.com<br />
support@johncompanies.com<br />
<br />
cat > /usr/local/www/wiki/.htaccess<br />
AuthType Basic<br />
AuthUserFile /usr/local/etc/apache22/wiki.passwd<br />
AuthName wiki<br />
require valid-user<br />
satisfy any<br />
<br />
cd /usr/local/etc/apache22<br />
htpasswd -c wiki.passwd admin<br />
<br />
https://69.55.229.8/index.php<br />
<br />
use mysql (innodb)<br />
wiki name: JCWiki<br />
Support / (mail pass) / support@johncompanies.com<br />
<br />
cat > /usr/local/www/wiki/LocalSettings.php<br />
<br />
<?php<br />
# This file was automatically generated by the MediaWiki 1.19.2<br />
# installer. If you make manual changes, please keep track in case you<br />
# need to recreate them later.<br />
#<br />
# See includes/DefaultSettings.php for all configurable settings<br />
# and their default values, but don't forget to make changes in _this_<br />
# file, not there.<br />
#<br />
# Further documentation for configuration settings may be found at:<br />
# http://www.mediawiki.org/wiki/Manual:Configuration_settings<br />
<br />
# Protect against web entry<br />
if ( !defined( 'MEDIAWIKI' ) ) {<br />
exit;<br />
}<br />
<br />
## Uncomment this to disable output compression<br />
# $wgDisableOutputCompression = true;<br />
<br />
$wgSitename = "JCWiki";<br />
<br />
## The URL base path to the directory containing the wiki;<br />
## defaults for all runtime URL paths are based off of this.<br />
## For more information on customizing the URLs please see:<br />
## http://www.mediawiki.org/wiki/Manual:Short_URL<br />
$wgScriptPath = "";<br />
$wgScriptExtension = ".php";<br />
<br />
## The protocol and server name to use in fully-qualified URLs<br />
$wgServer = "https://69.55.229.8";<br />
<br />
## The relative URL path to the skins directory<br />
$wgStylePath = "$wgScriptPath/skins";<br />
<br />
## The relative URL path to the logo. Make sure you change this from the default,<br />
## or else you'll overwrite your logo when you upgrade!<br />
#$wgLogo = "$wgStylePath/common/images/wiki.png";<br />
$wgLogo = "$wgStylePath/common/images/jclogo.gif";<br />
<br />
## UPO means: this is also a user preference option<br />
<br />
$wgEnableEmail = true;<br />
$wgEnableUserEmail = true; # UPO<br />
<br />
$wgEmergencyContact = "apache@69.55.229.8";<br />
$wgPasswordSender = "apache@69.55.229.8";<br />
<br />
$wgEnotifUserTalk = false; # UPO<br />
$wgEnotifWatchlist = false; # UPO<br />
$wgEmailAuthentication = true;<br />
<br />
## Database settings<br />
$wgDBtype = "mysql";<br />
$wgDBserver = "localhost";<br />
$wgDBname = "my_wiki";<br />
$wgDBuser = "root";<br />
$wgDBpassword = "";<br />
<br />
# MySQL specific settings<br />
$wgDBprefix = "";<br />
<br />
# MySQL table options to use during installation or update<br />
$wgDBTableOptions = "ENGINE=InnoDB, DEFAULT CHARSET=binary";<br />
<br />
# Experimental charset support for MySQL 5.0.<br />
$wgDBmysql5 = false;<br />
<br />
## Shared memory settings<br />
$wgMainCacheType = CACHE_NONE;<br />
$wgMemCachedServers = array();<br />
<br />
## To enable image uploads, make sure the 'images' directory<br />
## is writable, then set this to true:<br />
$wgEnableUploads = false;<br />
#$wgUseImageMagick = true;<br />
#$wgImageMagickConvertCommand = "/usr/bin/convert";<br />
<br />
# InstantCommons allows wiki to use images from http://commons.wikimedia.org<br />
$wgUseInstantCommons = false;<br />
<br />
## If you use ImageMagick (or any other shell command) on a<br />
## Linux server, this will need to be set to the name of an<br />
## available UTF-8 locale<br />
$wgShellLocale = "en_US.utf8";<br />
<br />
## If you want to use image uploads under safe mode,<br />
## create the directories images/archive, images/thumb and<br />
## images/temp, and make them all writable. Then uncomment<br />
## this, if it's not already uncommented:<br />
#$wgHashedUploadDirectory = false;<br />
<br />
## Set $wgCacheDirectory to a writable directory on the web server<br />
## to make your wiki go slightly faster. The directory should not<br />
## be publically accessible from the web.<br />
#$wgCacheDirectory = "$IP/cache";<br />
<br />
# Site language code, should be one of the list in ./languages/Names.php<br />
$wgLanguageCode = "en";<br />
<br />
$wgSecretKey = "abc699ef26890b49b4055430f8ebbd25e84cce21a7e53aeaec4d4313af4c9739";<br />
<br />
# Site upgrade key. Must be set to a string (default provided) to turn on the<br />
# web installer while LocalSettings.php is in place<br />
$wgUpgradeKey = "3196710f4a7d7332";<br />
<br />
## Default skin: you can change the default skin. Use the internal symbolic<br />
## names, ie 'standard', 'nostalgia', 'cologneblue', 'monobook', 'vector':<br />
$wgDefaultSkin = "vector";<br />
<br />
## For attaching licensing metadata to pages, and displaying an<br />
## appropriate copyright notice / icon. GNU Free Documentation<br />
## License and Creative Commons licenses are supported so far.<br />
$wgRightsPage = ""; # Set to the title of a wiki page that describes your license/copyright<br />
$wgRightsUrl = "";<br />
$wgRightsText = "";<br />
$wgRightsIcon = "";<br />
<br />
# Path to the GNU diff3 utility. Used for conflict resolution.<br />
$wgDiff3 = "/usr/bin/diff3";<br />
<br />
# Query string length limit for ResourceLoader. You should only set this if<br />
# your web server has a query string length limit (then set it to that limit),<br />
# or if you have suhosin.get.max_value_length set in php.ini (then set it to<br />
# that value)<br />
$wgResourceLoaderMaxQueryLength = -1;<br />
<br />
<br />
<br />
# End of automatically generated settings.<br />
# Add more configuration options below.<br />
<br />
<br />
</pre><br />
<br />
== copy/backup wiki ==<br />
on main/primary wiki:<br />
<pre><br />
/usr/local/etc/rc.d/mysql-server stop<br />
ssh 69.55.230.18 "/etc/init.d/mysql stop"<br />
rsync -av /var/db/mysql/my_wiki/ 69.55.230.18:/var/lib/mysql/my_wiki/<br />
rsync -av /var/db/mysql/ib* 69.55.230.18:/var/lib/mysql/<br />
/usr/local/etc/rc.d/mysql-server start<br />
ssh 69.55.230.18 "/etc/init.d/mysql start"<br />
<br />
</pre></div>75.80.128.147https://wiki.jcihosting.com/index.php?title=Infrastructure_Machines&diff=2212Infrastructure Machines2020-03-17T18:24:23Z<p>75.80.128.147: /* virt11 */</p>
<hr />
<div>= jails =<br />
<br />
== jail1 ==<br />
<br />
* Location: castle, SHUTDOWN<br />
* OS: FreeBSD 6.2 i386<br />
* Networking: Priv IP: 10.1.4.101 (PCI nic), Pub IP: 69.55.230.107 (onboard)<br />
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.<br />
* Remote management: none<br />
* Disk accounting: gvinum<br />
<br />
== jail2 ==<br />
<br />
* Location: castle, cab 6-16<br />
* OS: FreeBSD 7.2 amd64<br />
* Networking: Priv IP: 10.1.4.102, Pub IP: 69.55.228.53 (2 onboard nics)<br />
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: one 146 GB (2 x 146GB) RAID1 array, two 300 GB (4 x 300GB) RAID1 arrays running on an LSI-based, Dell-branded (PERC 6/i) RAID card.<br />
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.232 <br />
* Disk accounting: md<br />
<br />
== jail3 ==<br />
* Location: I2b SHUTDOWN<br />
* OS: FreeBSD 8.3 amd64<br />
* Networking: Priv IP: 10.1.2.103, Pub IP: 69.55.229.7 (2 onboard nics)<br />
* Hardware: Supermicro (custom build). 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: two 300 GB (2 x 300GB) RAID1 array running on a 3ware 8006-2LP RAID card.<br />
* Remote management: none<br />
* Disk accounting: md<br />
<br />
=== Notes ===<br />
* '''We should not add users to this server since it is at I2B'''<br />
* must be ssh'd to from nat2<br />
* is a super jail for customer col01737<br />
<br />
== jail4 ==<br />
<br />
* Location: castle, cab 6-17<br />
* OS: FreeBSD 9.1 x86_64<br />
* Networking: Priv IP: 10.1.4.104, Pub IP: 69.55.228.104 (2 onboard nics) <br />
* Hardware: Dell 2850. 6 x 300GB SCSI drives (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.<br />
* CPU: 2 x Intel(R) Xeon(TM) CPU 2.80GHz (8 virtual CPUs)<br />
* RAM: 16 GB ( 4 x 4GB Reg ECC )<br />
* Drives: one 1.4 TB RAID 5 array (6 x 300GB SCSI) Dell-branded (PERC 4e)LSI megarc RAID card.<br />
* Remote management: None <br />
* Disk accounting: md<br />
<br />
<br />
=== Notes ===<br />
Only FreeBSD 9.1 jail<br />
Not upgraded to FBSD 9.2 or 9.3 because too many libraries modified (would require customers to rebuild apps).<br />
<br />
== jail5 ==<br />
<br />
* Location: castle, cab 3-6<br />
* OS: FreeBSD 10.1 x86_64<br />
* Networking: Priv IP: 10.1.4.105, Pub IP: 69.55.230.105 (2 onboard nics) <br />
* Hardware: Supermicro JC-14004 - Intel S1200BTL motherboard - 6 SATA/SAS drive bays (2 colums of 3), Dual power supply.<br />
* CPU: 1 x Intel(R) Xeon(TM) E3-1230 V2 CPU 3.30 GHz (8 virtual CPUs)<br />
* RAM: 32 GB ( 4 x 8GB ECC )<br />
* Drives: 1x80 GB SATA SSD on motherboard + one 2.6 TB RAID 5 array 4x1 TB + 3ware 9650 RAID card.<br />
* Remote management: Intel RMM 4 - 10.1.4.235<br />
* Disk accounting: md<br />
<br />
=== Notes ===<br />
Only FreeBSD 10.1 jail used for bhyve virtuals.<br />
<br />
Use ~+Ctrl-D to disconnect from console (vm attach colXXXXX).<br />
<br />
== jail6 ==<br />
<br />
* Location: castle, cab 6-16<br />
* OS: FreeBSD 10.3 x86_64<br />
* Networking: Priv IP: 10.1.4.106, Pub IP: 69.55.230.106 (2 onboard nics) <br />
* Hardware: Supermicro JC-14004 - Intel S1200BTL motherboard - 6 SATA/SAS drive bays (2 colums of 3), Dual power supply.<br />
* CPU: 1 x Intel(R) Xeon(TM) E3-1230 V2 CPU 3.30 GHz (8 virtual CPUs)<br />
* RAM: 32 GB ( 4 x 8GB ECC )<br />
* Drives: one 2.7 TB ZFS RAID 6 array 5x1 TB <br />
* Remote management: Intel RMM 4 - 10.1.4.236<br />
* Disk accounting: zfs<br />
<br />
== jail7 ==<br />
<br />
* Location: castle, cab 3-5 SCHEUDLED SHUTDOWN 9/30/19<br />
* OS: FreeBSD 6.3 i386<br />
* Networking: Priv IP: 10.1.4.107, Pub IP: 69.55.230.108 (2 onboard nics)<br />
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: two 146 GB (4 x 146GB) RAID1 arrays, one 74 GB (2 x 74GB) RAID1 array running on an LSI-based, Dell-branded (PERC 6/i) RAID card.<br />
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.237<br />
* Disk accounting: gvinum<br />
<br />
=== Notes ===<br />
Do not run a verify while OS/jails running, will crash.<br />
<br />
== jail8 ==<br />
<br />
* Location: castle, cab 3-6<br />
* OS: FreeBSD 8.0 amd64<br />
* Networking: Priv IP: 10.1.4.108, Pub IP: 69.55.234.2 (2 onboard nics)<br />
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 300 GB (2 x 300GB) RAID1 array, one 400 GB (2 x 400GB) RAID1 array, running on an LSI-based, Dell-branded (PERC 6/i) RAID card.<br />
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.238<br />
* Disk accounting: md<br />
<br />
<br />
== jail9 ==<br />
<br />
* Location: castle, cab 3-6<br />
* OS: FreeBSD 8.2 amd64<br />
* Networking: Priv IP: 10.1.4.109, Pub IP: 69.55.232.36 (2 onboard nics)<br />
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 400 GB (2 x 300GB) RAID1 array running on an LSI-based, Dell-branded (PERC 5/i) RAID card.<br />
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.239<br />
* Disk accounting: md<br />
<br />
<br />
== jail11 ==<br />
<br />
* Location: castle, cab 3-7<br />
* OS: FreeBSD 4.7 i386<br />
* Networking: Priv IP: 10.1.4.111 (PCI nic), Pub IP: 69.55.236.92 (onboard)<br />
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.<br />
* Remote management: none<br />
* Disk accounting: vinum<br />
<br />
<br />
== mx1 ==<br />
<br />
* Location: castle, SHUTDOWN AND SCRAPPED<br />
* OS: FreeBSD 4.11 i386<br />
* Networking: Priv IP: 10.1.4.201 (PCI nic), Pub IP: 69.55.237.3 (onboard)<br />
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: one 36 GB (2 x 36GB) RAID1 array, one 74 GB (2 x 74GB) RAID1 array running on an Adaptec-based, Dell-branded (perc) RAID card.<br />
* Remote management: none<br />
* Disk accounting: vinum<br />
<br />
=== Notes ===<br />
* is our (old) backup mail/dns vps service host<br />
<br />
== mx2 ==<br />
<br />
* Location: castle, SHUTDOWN AND SCRAPPED<br />
* OS: FreeBSD 7.1 i386<br />
* Networking: Priv IP: 10.1.4.202 (PCI nic), Pub IP: 69.55.237.90 (onboard)<br />
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.<br />
* Disk accounting: gvinum<br />
<br />
=== Notes ===<br />
* is our latest backup mail/dns vps service host<br />
<br />
== jail17 ==<br />
* Location: castle, cab 3-7<br />
* OS: FreeBSD 4.10 i386<br />
* Networking: Priv IP: 10.1.4.117 (PCI nic), Pub IP: 69.55.228.2 (onboard nics)<br />
* Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: two 74 GB (4 x 74GB) RAID1 arrays, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S RAID card.<br />
* Remote management: none<br />
* Disk accounting: vinum<br />
* Host of devweb.johncompanies.com and www.utopian.com/mail.utopian.com<br />
<br />
== jail18 ==<br />
* Location: castle, cab 3-5 SCHEDULED SHUTDOWN 9/30/19<br />
* OS: FreeBSD 4.10 i386<br />
* Networking: Priv IP: 10.1.4.118 (PCI nic), Pub IP: 69.55.228.2 (onboard nics)<br />
* Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: two 74 GB (4 x 74GB) RAID1 arrays, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S RAID card.<br />
* Remote management: none<br />
* Disk accounting: vinum<br />
* Host of ns2c.johncompanies.com (now on ns2c.johncompanies.com on ganeti)<br />
<br />
== jail19 ==<br />
* Location: castle, cab 3-5 SCHEDULED SHUTDOWN 9/30/19<br />
* OS: FreeBSD 6.1 i386<br />
* Networking: Priv IP: 10.1.4.119 (PCI nic), Pub IP: 69.55.228.200 (onboard nics)<br />
* Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: one 74 GB (2 x 74GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S RAID card.<br />
* Remote management: none<br />
* Disk accounting: gvinum<br />
<br />
= virts =<br />
<br />
== quar1 ==<br />
<br />
* Location: castle, SHUTDOWN AND SCRAPPED<br />
* OS: RedHat 7.3 x86<br />
* Networking: Priv IP: 10.1.4.151 (PCI nic), Pub IP: 69.55.227.2 (onboard nic)<br />
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: one 36 GB (2 x 36GB) RAID1 array, one 74 GB (2 x 74GB) RAID1 array, running on an Adaptec-based, Dell-branded (perc) RAID card.<br />
* Remote management: none<br />
* Virtuozzo version: 2.6.1<br />
* VZ license: hwid=23C0.C0E1.6FDD.08BA.8971.8E1C.EBD5.1EDC serial=0DE6.903E.E239.E23F.470C.4369.4104.A5A4<br />
<br />
=== Notes ===<br />
* used to be the home of customers who's VE's would just run out of control/badly<br />
* has a max of 10 VE's allowed to run<br />
<br />
== virt9 ==<br />
<br />
* Location: castle, cab 3-7<br />
* OS: RedHat 7.3 x86<br />
* Networking: Priv IP: 10.1.4.59 (PCI nic), Pub IP: 69.55.226.161 (onboard nic)<br />
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: two 74 GB (2 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.<br />
* Remote management: none<br />
* Virtuozzo version: 2.6.1<br />
* VZ license: hwid=BC15.B4D6.0D25.A5FE.F3BA.D518.E351.AE3F serial=F6AD.B6B4.5650.8869.C97C.73EE.AF65.FA8B<br />
<br />
<br />
== virt11 ==<br />
<br />
* Location: castle, cab 3-6<br />
* OS: CentOS 5.4 x86<br />
* Networking: Priv IP: 10.1.4.61, Pub IP: 69.55.238.3, 2 onboard nics<br />
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 400 GB (2 x 400GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.<br />
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.211<br />
* Virtuozzo version: 4.0.0<br />
* VZ license: hwid="029D.A187.78E1.480F.49E3.E20A.7389.7F79" serial="163C.F3E2.195F.96B5.2D38.8937.9600.4A05"<br />
key_number="VZ.00172378.0006"<br />
<br />
== virt12 ==<br />
<br />
* Location: castle, cab 3-7<br />
* OS: CentOS 5.2 x86<br />
* Networking: Priv IP: 10.1.4.62, Pub IP: 69.55.227.70, 2 onboard nics<br />
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: two 300 GB (2 x 300GB) RAID1 arrays one 400 GB (2 x 400GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.<br />
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.212<br />
* Virtuozzo version: 4.0.0<br />
* VZ license: hwid="0C53.A413.E095.B4F4.51BC.D740.6919.A77B" serial="84E5.9498.3759.E683.E24B.2514.CA72.DC31"<br />
<br />
== virt13 ==<br />
<br />
* Location: castle, cab 6-17<br />
* Switch port: P13-<br />
* OS: CentOS 6.2 x86_64<br />
* Networking: Priv IP: 10.1.4.63, Pub IP: 69.55.226.2, 2 onboard nics<br />
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* CPU: 2 x Intel(R) Xeon(R) CPU E5420 @ 2.50GHz (8 virtual cores)<br />
* RAM: 32 GB (8 x 4GB DDR2 FB-DIMM ECC 667MHz)<br />
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 600 GB (2 x 600GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.<br />
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.213<br />
* Virtuozzo version: 4.7.0<br />
* VZ license: hwid="7D07.93BE.0B1F.7D2B.B039.4B5B.48B6.453B" serial="60A4.A94C.44BB.DCD6.8D03.1778.605B.10FE"<br />
<br />
=== Notes ===<br />
* home to our latest/current signups<br />
* currently the only 64bit vz host<br />
<br />
== virt14 ==<br />
<br />
* Location: castle, cab 6-16<br />
* Switch Port: p13-<br />
* OS: CentOS 6.4 x86_64<br />
* Networking: Priv IP: 10.1.4.64 Pub IP: 69.55.225.14 2 onboard nics<br />
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.<br />
* CPU: 2 x Xeon 5140 Dual Core @ 2.33GHz (4 virtual CPUs)<br />
* RAM: 32 GB (8 x 4GB Reg ECC)<br />
* Drives: one 146 GB (2 x 146 GB SAS) RAID1 array, and one 1TB RAID1 array (2 x 1 TB SATA), running on an LSI-based, Dell-branded (perc 5/i) RAID card.<br />
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.214<br />
* Virtuozzo version: 4.7.0<br />
* VZ license: hwid="" serial=""<br />
<br />
=== Notes ===<br />
* our latest virt <br />
* Temp server to offload Virt13 till we can get a Cloud going.<br />
* virt 13 and 14 currently the only 64bit vz hosts<br />
<br />
== virt15 ==<br />
<br />
* Location: SHUTDOWN<br />
* OS: RedHat 9 x86<br />
* Networking: Priv IP: 10.1.4.65, Pub IP: 69.55.232.160 (2 onboard nics)<br />
* Hardware: Supermicro (custom build). 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: one 74 GB (2 x 74GB) RAID1 array, two 146 GB (2 x 146GB) RAID1 arrays, running on an LSI MegaRAID SCSI 320-1 RAID card.<br />
* Remote management: none<br />
* Virtuozzo version: 2.6.2<br />
* VZ license: hwid=A90F.6F48.E723.D8BA.3025.184A.5B73.D11E serial=E94B.5164.C1E6.A67F.67D1.7D96.0B6C.5524<br />
<br />
== virt16 ==<br />
<br />
* Location: castle, cab 3-7<br />
* OS: Fedora Core 4 x86<br />
* Networking: Priv IP: 10.1.4.66, Pub IP: 69.55.232.2 (2 onboard nics)<br />
* Hardware: Supermicro (custom build). 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: one 74 GB (2 x 74GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an LSI MegaRAID SCSI 320-1 RAID card.<br />
* Remote management: none<br />
* Virtuozzo version: 3.0.0<br />
* VZ license: hwid=DEFA.A325.7230.BBC8.9715.8B52.3FD7.27BE serial=66C0.41EA.3FBB.11D3.9CC6.55C7.09AE.14AB<br />
<br />
<br />
== virt17 ==<br />
<br />
* Location: castle, cab 3-6<br />
* OS: CentOS 4.4 x86<br />
* Networking: Priv IP: 10.1.4.67, Pub IP: 69.55.232.162, 2 onboard nics<br />
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: two 146 GB (2 x 146GB) RAID1 arrays running on an LSI-based, Dell-branded (perc 5/i) RAID card.<br />
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.217<br />
* Virtuozzo version: 3.0.0<br />
* VZ license: hwid=2E14.AED9.70B8.C26E.D99F.B0D3.BCD2.229C serial=2A11.DAD0.61DB.E889.8DF4.9AF7.CF82.3C37<br />
<br />
<br />
== virt19 ==<br />
<br />
* Location: castle, cab 3-6<br />
* OS: CentOS 5.2 x86<br />
* Networking: Priv IP: 10.1.4.69, Pub IP: 69.55.236.2, 2 onboard nics<br />
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 300 GB (2 x 300GB) RAID1 array, running on an LSI-based, Dell-branded (perc 5/i) RAID card.<br />
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.219<br />
* Virtuozzo version: 3.0.0<br />
* VZ license: hwid=3968.13F7.B2AC.8952.8E19.13A9.6EF5.5822 serial=061D.84CD.CCE5.B213.15B5.C061.D6A7.B034<br />
<br />
= mail =<br />
== Summary ==<br />
This machine (mail) is the swiss army knife of the company, playing host to many services and functions. <br />
<br />
* Location: castle, cab 3-7<br />
* OS: FreeBSD 4.10 x86<br />
* Networking: Priv IP: 10.1.4.5, Pub IPs: 69.55.230.2, 69.55.225.225 (ns1c jail), 69.55.230.9. 1 onboard and 1 PCI<br />
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.<br />
<br />
== Services Provided ==<br />
* mail<br />
* web<br />
* mysql<br />
* bigbrother server/pager<br />
* snmp<br />
* named in a jail (ns1c)<br />
<br />
== email ==<br />
This server hosts mail for johncompanies.com (mail.johncompanies.com). Sendmail 8.13.6/8.13.6 is listening on 69.55.230.2 port 25 for incoming mail. Relaying is allowed per /etc/mail/relay-domains<br />
Other addresses (aliases) are defined per /etc/mail/aliases<br />
<br />
The following active users have mail hosted on this server:<br />
* dave<br />
* linux<br />
* support<br />
* payments<br />
* sales<br />
* tech1<br />
* info<br />
<br />
Traditionally, mail is checked via shell apps (pine). qpopper (pop3s) is running to allow mail downloading. Checking mail in this way causes an opened INBOX in pine to lock read-only. For this reason, we tee incoming mail to support and linux to tech1.<br />
<br />
Procmail rules are setup to filter spam and send text messages. They are enabled for info, support, linux, tech1, dave and can be found in ~/Procmail/, for example:<br />
<pre># more ~support/Procmail/rc.emergency<br />
:0c # use c only if you want to forward a copy and file the original later<br />
* ^Subject:.*\<emergency\><br />
* ! ^Subject:\<re\><br />
{<br />
:0h<br />
FROMANDSUBJECT=|formail -XFrom: -XSubject:<br />
<br />
:0fwh<br />
| /usr/local/bin/formail -I"Subject: " -I"To: pager@johncompanies.com" ; echo $FROMANDSUBJECT ; echo<br />
<br />
:0<br />
! -t<br />
}<br />
<br />
</pre><br />
<br />
control: <tt>cd /etc/mail; make stop</tt> (stop), <tt>cd /etc/mail; make start</tt> (start)<br />
<br />
The following aliases are also in place:<br />
<br />
<pre>debian: linux<br />
jobs: info<br />
careers: info<br />
#reboot: 6128102202@txt.att.net<br />
#reboot: 8582298897@vtext.com<br />
reboot: pager<br />
#pager: 8582298897@vtext.com<br />
pager: 4158718324@txt.att.net<br />
tech1on: "| /usr/local/sbin/tech1on.sh"<br />
tech1off: "| /usr/local/sbin/tech1off.sh"</pre><br />
<br />
To change them, edit <tt>/etc/aliases</tt> and then run <tt>newaliases</tt><br />
<br />
Note on tech1: this address was setup as a read-only address to be mirrored on all email coming into support and linux. We set this up so we could easily check support mail via a pop client- popping email locks out the user in pine so checking support/linux directly via pop was not an option. When checking and responding to email that comes into tech1, care should be taken to make sure it is sent as/under an address other than tech1. This is cause tech1 is not monitored by support staff as closely as email to support/linux. Further, the tech on call may not be checking tech1. Lastly, because of the nature of the copying, you will sometimes notice certain automated email/notices are received 2x in support- this is because of/related to the tech1 mirror.<br />
<br />
To enable it (on mail, run):<br />
<br />
<tt>~support/tech1on.sh</tt><br />
<br />
To disable<br />
<br />
<tt>~support/tech1off.sh</tt><br />
<br />
Or via email:<br />
<br />
<tt>tech1on@johncompanies.com<br />
tech1off@johncompanies.com</tt><br />
<br />
== IP Blocking ==<br />
<br />
<pre><br />
01000 deny ip from 188.92.72.5 to any<br />
01003 deny ip from any to 122.49.31.50<br />
01004 deny ip from 122.49.31.50 to any<br />
01014 deny ip from 74.208.225.225 to any<br />
01015 deny ip from any to 216.243.118.35<br />
01016 deny ip from 216.243.118.35 to any<br />
01017 deny ip from any to 216.243.118.36<br />
01018 deny ip from 216.243.118.36 to any<br />
01020 deny ip from 112.215.0.0/18 to any 2014-08-13 Blocked PT Excelcomindo Pratama (Indonesia) for fradulent credit card attempts <br />
01020 deny ip from 112.215.64.0/20 to any 2014-08-13 Blocked PT Excelcomindo Pratama (Indonesia) for fradulent credit card attempts <br />
01022 deny ip from 120.168.0.0/24 to any 2014-08-13 Blocked Indosat 3G Broadband (Indonesia) for fradulent credit card attempts<br />
01022 deny ip from 120.175.213.0/24 to any 2014-08-13 Blocked Indosat 3G Broadband (Indonesia) for fradulent credit card attempts<br />
<br />
</pre><br />
<br />
== web == <br />
<br />
See [[Management_System_/_Public_Website_/_Signup|Management System / Public Website / Signup]]<br />
<br />
== mysql ==<br />
mysql 4.1.22 is running on port 3306<br />
<br />
* datadir: <tt>/mnt/data1/db/mysql/</tt><br />
* config: <tt>/etc/my.cnf</tt><br />
* database: <tt>jc</tt><br />
* control: <tt>/usr/local/etc/rc.d/mysql-server.sh stop</tt> (stop), <tt>/usr/local/etc/rc.d/mysql-server.sh start</tt> (start)<br />
<br />
== bigbrother ==<br />
There is a client running on mail (which monitors the services running on mail and mail itself), installed under <tt>/usr/home/bb/bbc1.9e-btf</tt><br><br />
And the big brother pager/server (which displays information gathered from all bb-monitored machines, including mail) is installed under <tt>/usr/home/bb/bbsrc/bb1.9i-btf</tt><br />
<br />
Both are running under the user <tt>bb</tt><br />
<br />
Refer to [[BigBrother]] for more about use.<br />
<br />
== DNS (ns1c.johncompanies.com) ==<br />
ns1c is a jail running on the mail server, who's IP is 69.55.225.225<br />
<br />
It's running from <tt>/mnt/data1/ns1c-dir</tt><br />
<br />
See [[DNS]] for more details<br />
<br />
== Usage and Notes ==<br />
* always mounted to backup1 and backup2 via nfs: <br />
<pre>backup2:/mnt/data1 on /backup (nfs)<br />
backup2:/mnt/data2 on /backup2 (nfs)<br />
backup2:/mnt/data3 on /backup3 (nfs)<br />
backup2:/mnt/data4 on /backup4 (nfs)<br />
backup1:/data on /backup1 (nfs)<br />
</pre><br />
<br />
== Cronjobs ==<br />
* * * * * /usr/local/www/mgmt/mrtg/mrtg.sh > /dev/null 2>&1<br />
Gathers up data for our mrtg/load graphs<br />
<br />
*/5 * * * * /usr/local/bin/rsync -a root@nat2:/mnt/data1/mrtg/data/ /usr/local/www/mgmt/mrtg/data/<br />
Gathers up data from i2b servers for our mrtg/load graphs<br />
<br />
40 0 * * * /usr/local/bin/rsync -a root@nat2:"/mnt/data1/mrtg/*.cfg" /usr/local/www/mgmt/mrtg<br />
Gathers up mrtg configuration (port names) from i2b switches for our mrtg/load graphs<br />
<br />
41 0 * * * for f in `grep -l "mnt\/data1" /usr/local/www/mgmt/mrtg/switch-p*.cfg`; do cat $f | sed s#\/mnt\/data1#\/usr\/local\/www\/mgmt# > $f.new; mv $f.new $f; done<br />
Gathers up mrtg configuration (port names) from castle switches for our mrtg/load graphs<br />
<br />
1 0 1 * * cp /usr/local/www/mgmt/html/top20ip /usr/local/www/mgmt/html/top20ip_last<br />
1 0 1 * * cp /usr/local/www/mgmt/html/top20customers /usr/local/www/mgmt/html/top20customers_last<br />
2 * * * * /usr/local/www/cronjobs/top20ip.pl > /dev/null 2>&1<br />
15 * * * * /usr/local/www/cronjobs/top20customer.pl > /dev/null 2>&1<br />
1 0 1 * * rm /usr/local/www/mgmt/html/bandtrack<br />
Archiving and generation of bandwidth statistics presented in mgmt -> Reference -> Bandwidth<br />
<br />
1 0 * * * /usr/local/etc/rsync.backup<br />
Nightly backup script<br />
<br />
0 1 * * * /usr/local/www/mgmt/awstats/wwwroot/cgi-bin/awstats.pl -config=jcpub -update<br />
Public web traffic stats<br />
<br />
15 0 * * * rm /usr/local/www/mgmt/bwgraphs/*.png<br />
16 0 * * * rm /usr/local/www/am/bwgraphs/*<br />
Cleanup for graph-related temp data generated by customers using the bandwidth reports via the AM <br />
<br />
10 0 1 * * /usr/local/www/cronjobs/monthly_bandwidth_report.pl<br />
Monthly bandwidth overage report<br />
<br />
*/3 * * * * /usr/local/www/cronjobs/bbcheck.pl<br />
Updates mgmt with bb monitoring issues<br />
<br />
5 0 * * * /usr/local/www/cronjobs/shutdownreminder.pl<br />
Emails customers reminding them of upcoming shutdown date<br />
<br />
7 0 * * * /usr/local/www/cronjobs/invoice_email.pl<br />
Emails customers who have invoices and are set to auto-email (currently no customer gets these)<br />
<br />
8 */4 * * * /usr/local/www/cronjobs/mysqlrepchk.pl<br />
Checking that we are properly replicating (mysql) traffic data from bwdb to backup1<br />
<br />
16 0 1 * * /usr/local/www/cronjobs/purge_traffic.pl<br />
Removed old traffic data from the traffic database (running on backup1)<br />
<br />
*/5 * * * * chmod 0700 /usr/local/www/ccard_orders/* && mv /usr/local/www/ccard_orders/* /usr/local/www/ccard_orders/done<br />
Secure credit card data: set root-read-only <br />
<br />
25 0 * * * /usr/local/www/cronjobs/biller.pl<br />
Enters service charges in customer billing ledgers<br />
<br />
10 13 * * * /usr/local/www/cronjobs/pfp_batch_gather.pl<br />
Looks for customers with balance due and active credit card on file, prepares a payflow batch<br />
<br />
10 14 * * * /usr/local/www/cronjobs/pfp_batch_process.pl<br />
Tries to collect ccard funds for items in payflow batch - communicates with payflow<br />
<br />
15 13 * * * /usr/local/www/cronjobs/pb_batch_gather.pl<br />
Looks for customers with balance due and active paypal billing agreement on file, prepares a paypal batch<br />
<br />
15 14 * * * /usr/local/www/cronjobs/pb_batch_process.pl<br />
Tries to collect paypal funds for items in paypal batch - communicates with paypal<br />
<br />
0 7 * * 1 /usr/local/www/cronjobs/email_pmt_reminder.pl<br />
Emails customers in arrears, reminding them to pay<br />
<br />
0 0 1 * * /usr/bin/mail -s 'archive sent mail in pine' support@johncompanies.com < /dev/null<br />
Reminds us to archive sent mail<br />
<br />
0 3 * * * /usr/local/bin/rsync -a isys.e-monitoring.net:/var/mail /backup2/isys; /usr/local/bin/rsync -a isys.e-monitoring.net:/usr/home /backup2/isys<br />
Backup data on isys<br />
<br />
== Regular maintenance ==<br />
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]<br />
<br />
== Building a new Mail Server ==<br />
<br />
=== Installations ===<br />
<br />
I used FreeBSD 11.2 <br />
<br />
The order is important especially for the Web Server.<br />
<br />
==== Web Server ====<br />
<br />
I used FreeBSD 11.2<br />
perl 5.26 <br />
OpenSSL 1.0.2o-freebsd<br />
pcre<br />
apache22 <br />
mod_perl2<br />
PayflowPro<br />
mariadb 55 server and client<br />
<br />
Installation order is important<br />
<br />
install perl 5.26.2 from ports<br />
<br />
<ore><br />
cd /usr/ports/lang/perl5.26/<br />
make <br />
[X] PERL_64BITINT Use 64 bit integers (on i386)<br />
[X] USE_PERL Rewrite links in /usr/bin<br />
(the rest unchecked<br />
make install<br />
</pre><br />
<br />
install OpenSSL 1.0.2o-freebsd<br />
<br />
<pre><br />
cd /usr/ports/<br />
make install<br />
</pre><br />
<br />
install pcre<br />
<br />
<pre><br />
cd /usr/ports/<br />
make install<br />
</pre><br />
<br />
<br />
install Apache22<br />
<br />
<pre><br />
cd /usr/ports/distfiles<br />
fetch http://archive.apache.org/dist/httpd/httpd-2.2.32.tar.gz<br />
<br />
<br />
cd /usr/ports/www/apache22/tmp<br />
fetch --no-verify-peer http://mirror.nexcess.net/apache//httpd/httpd-2.2.34.tar.gz<br />
tar xvzf httpd-2.2.34.tar.gz<br />
./configure --prefix=/usr/local/apache --with-ssl=/usr/local/openssl/ --enable-ssl --enable-so --with-mpm=prefork --enable-threads --enable-mods-shared='mime alias setenvif dir' --enable-modules='mime alias setenvif dir' --with-pcre=/usr/local<br />
make install<br />
apachectl restart<br />
<br />
cd /usr/ports/www/apache22<br />
echo "DEFAULT_VERSIONS+=apache=2.2" >> /etc/make.conf<br />
make DISABLE_VULNERABILITIES=yes<br />
make install<br />
</pre><br />
<br />
<br />
install mod_perl2<br />
<br />
<pre><br />
cd /usr/ports/www/mod_perl2<br />
echo "DEFAULT_VERSIONS+=apache=2.2" >> /etc/make.conf<br />
make DISABLE_VULNERABILITIES=yes<br />
make install<br />
</pre><br />
<br />
<br />
install mariadb<br />
<br />
<pre><br />
cd /usr/ports/databases/mariadb-103-server<br />
echo "DEFAULT_VERSIONS+=apache=2.2" >> /etc/make.conf<br />
make DISABLE_VULNERABILITIES=yes<br />
make install<br />
<br />
<br />
</pre><br />
<br />
==== Mail Server ====<br />
<br />
I used Postfix for email<br />
<br />
==== DNS Server (ns1c.johncompanies.com) ====<br />
<br />
= ns2c =<br />
== Summary ==<br />
<br />
* Location: castle, on lamphost ganeti cloud<br />
* OS: FreeBSD 11.2 x86_64<br />
* Networking: Pub IP: 69.55.230.3 Private access: gnt-instance console ns2c.johncompanies.com<br />
* Hardware: on ganeti cloud gn6.jcihosting.net secondary gn1.jcihosting.net<br />
* CPU: 1<br />
* RAM: 1 GB<br />
* Drives: 10 GB<br />
* Remote management: gnt-instance console ns2c.johncompanies.com<br />
<br />
= nat =<br />
== Summary ==<br />
This is the main machine to which we ssh and runs all our screen sessions. Further, it's ip runs in a special block which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.<br />
<br />
* Location: castle, cab 3-7<br />
* OS: FreeBSD 9.1 i386<br />
* Networking: Priv IP: 10.1.4.1, Pub IPs: 69.55.233.195, 69.55.233.196, 69.55.233.197, 69.55.233.198, 69.55.233.199. 1 onboard and 1 PCI<br />
* Hardware: Custom 1U. single power supply. <br />
* Drives: one 8 GB IDE drive<br />
<br />
== Services Provided ==<br />
* nat<br />
<br />
== nat control ==<br />
All rules are contained in and look like:<br />
<br />
<pre>cat /etc/ipnat.rules<br />
# www (was 69.55.230.12)<br />
# virt19<br />
#bimap fxp0 10.1.4.209/32 -> 69.55.233.198/32<br />
# virt18<br />
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32<br />
# virt13<br />
#bimap fxp0 10.1.4.213/32 -> 69.55.233.196/32<br />
# virt12<br />
#bimap fxp0 10.1.4.212/32 -> 69.55.233.196/32<br />
# virt17<br />
bimap fxp0 10.1.4.217/32 -> 69.55.233.196/32<br />
# virt11<br />
#bimap fxp0 10.1.4.211/32 -> 69.55.233.196/32<br />
# ASA<br />
#bimap fxp0 10.1.4.172/32 -> 69.55.233.196/32<br />
# P1A<br />
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32<br />
#bimap fxp0 10.1.4.238/32 -> 69.55.233.197/32<br />
# developer (was 69.55.230.17)<br />
# jail2<br />
#bimap fxp0 10.1.4.232/32 -> 69.55.233.198/32<br />
# jail8<br />
#bimap fxp0 10.1.4.238/32 -> 69.55.233.198/32<br />
# jail9<br />
#bimap fxp0 10.1.4.239/32 -> 69.55.233.198/32<br />
# POLL<br />
#BIMAP EM0 10.1.6.134/32 -> 69.55.230.20/32<br />
# 1U SUN<br />
#BIMAP EM0 10.1.4.4/32 -> 69.55.227.46/32<br />
# ??<br />
#BIMAP EM0 10.1.6.3/32 -> 69.55.230.100/32<br />
# random machine<br />
#bimap fxp0 10.1.6.13/32 -> 69.55.233.199/32<br />
#bimap fxp0 10.1.4.232/32 -> 69.55.233.199/32<br />
# OFFICE OUTBOUND TRAFFIC<br />
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp<br />
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32</pre><br />
<br />
A simple entry looks like:<br />
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32<br />
Which essentially means make private IP 10.1.4.240 reachable on 69.55.233.197 and allow 10.1.4.240 to communicate with the public internet via 69.55.233.197<br />
<br />
To reload new rule config:<br />
ipnat -C -F -f /etc/ipnat.rules<br />
<br />
You may want to setup natting, as above, when you need to reach a DRAC card's web interface, wherin the DRAC card only has a private IP.<br />
<br />
= nat2 =<br />
== Summary ==<br />
This is the main machine to which we ssh and runs all our screen sessions at i2b, and runs ns3c (this is kind of the what mail is to castle). Further, it's ip runs in IP space provided by i2b: 66.181.18.1 - 66.181.18.30, which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.<br />
<br />
* Location: i2b, cab 6<br />
* OS: FreeBSD 6.4 x86<br />
* Networking: Priv IP: 10.1.2.1, Pub IPs: 69.55.229.2, 69.55.229.3, 66.181.18.4, 66.181.18.5, 66.181.18.6, 66.181.18.7, 66.181.18.8, 66.181.18.9, 66.181.18.10, 66.181.18.11, 66.181.18.12, 66.181.18.13, 66.181.18.14 1 onboard and 1 PCI<br />
* Hardware: Custom 2U. 6 drive bays, non-hot-swappable. single power supply. <br />
* Drives: one 150 GB (2 x 150GB) RAID1 array running on a 3ware 8006 RAID card.<br />
<br />
== Services Provided ==<br />
* nat<br />
* bigbrother<br />
* ns3c (jail)<br />
* ntp<br />
<br />
== nat config ==<br />
Here's what's currently nat'd on nat2:<br />
<pre>cat /etc/ipnat.rules<br />
# sample entry<br />
#ATS-9<br />
bimap em0 10.1.2.79/32 -> 66.181.18.14/32<br />
#ATS-8<br />
bimap em0 10.1.2.78/32 -> 66.181.18.13/32<br />
#ATS-7<br />
bimap em0 10.1.2.77/32 -> 66.181.18.12/32<br />
#ATS-6<br />
bimap em0 10.1.2.76/32 -> 66.181.18.6/32<br />
#ATS-5<br />
bimap em0 10.1.2.75/32 -> 66.181.18.7/32<br />
#ATS-4<br />
bimap em0 10.1.2.74/32 -> 66.181.18.8/32<br />
#ATS-3<br />
bimap em0 10.1.2.73/32 -> 66.181.18.9/32<br />
#ATS-2<br />
bimap em0 10.1.2.72/32 -> 66.181.18.10/32<br />
#ATS-1<br />
bimap em0 10.1.2.71/32 -> 66.181.18.11/32<br />
#bwdb2<br />
bimap em0 10.1.2.4/32 -> 66.181.18.5/32<br />
<br />
# spare<br />
<br />
map em0 10.1.2.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp<br />
<br />
#bimap fxp0 10.1.6.49/32 -> 10.1.1.2/32<br />
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32<br />
</pre><br />
<br />
== build ==<br />
<br />
* partition map:<br />
<pre>/ 512m<br />
swap 1G<br />
/var 256m<br />
/tmp 256m<br />
/usr 5g<br />
/mnt/data1 ~</pre><br />
<br />
* edit /etc/make.conf <br />
<pre>echo "WITHOUT_X11=yes \<br />
KERNCONF=nat2 \<br />
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre><br />
<br />
* add settings to /boot/loader.conf and /boot.config<br />
<br />
<pre>echo "-Dh" >> /boot.config<br />
<br />
echo 'console="comconsole,vidconsole" \<br />
boot_multicons="YES" \<br />
boot_serial="YES" \<br />
comconsole_speed="115200"' >> /boot/loader.conf</pre><br />
<br />
* turn off all ttyv's except 0 and 1 in /etc/ttys<br />
also turn on ttyd0, change type to vt100:<br />
<pre>vi /etc/ttys<br />
<br />
ttyv2 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv3 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv4 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv5 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv6 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv7 "/usr/libexec/getty Pc" cons25 off secure<br />
# Serial terminals<br />
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.<br />
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure<br />
<br />
kill -1 1</pre><br />
<br />
on console server:<br />
vi /etc/remote<br />
(rename port to jail8 depending on where and which digi plugged into)<br />
test serial console<br />
<br />
<br />
* populate hosts<br />
<pre>echo "69.55.230.10 backup2" >> /etc/hosts<br />
echo "69.55.230.11 backup1" >> /etc/hosts<br />
echo "10.1.2.4 bwdb2" >> /etc/hosts<br />
echo "10.1.2.3 backup3" >> /etc/hosts</pre><br />
<br />
* put key in authorized_keys on backup3<br />
<pre>cd<br />
ssh-keygen -t dsa -b 1024</pre> <br />
(default location, leave password blank)<br />
<br />
<pre>cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys' <br />
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' <br />
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre> <br />
<br />
confirm that you can ssh to backup3 and backup 2 without getting a login prompt<br />
<br />
ssh backup3 hostname<br />
<br />
ssh backup2 hostname<br />
<br />
ssh backup1 hostname<br />
<br />
<br />
* edit root's path and login script:<br />
vi /root/.cshrc<br />
<br />
Change alias entries (add G):<br />
<pre>alias la ls -aG<br />
alias lf ls -FAG<br />
alias ll ls -lAG<br />
alias ls ls -AG<br />
alias mbm mb mount<br />
alias mbu mb umount</pre><br />
<br />
and alter the prompt, set the following:<br />
set prompt = "`/bin/hostname -s` %/# "<br />
<br />
* install cvsup<br />
<pre>cd /usr/ports/net/cvsup-without-gui <br />
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null</pre><br />
<br />
* get latest sources for this release:<br />
<pre>cd /usr/src <br />
echo "*default host=cvsup4.freebsd.org\<br />
*default base=/usr\<br />
*default prefix=/usr\<br />
*default release=cvs tag=RELENG_6_4\<br />
*default delete use-rel-suffix\<br />
*default compress\<br />
src-all" > sup<br />
<br />
cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null</pre><br />
<br />
<br />
* configure new kernel. <br />
<br />
cd /usr/src/sys/i386/conf <br />
scp backup2:/mnt/data4/build/freebsd/nat2-6.4 ./nat2<br />
<br />
* build, install kernel and world<br />
<br />
<pre>cd /boot<br />
<br />
mv kernel kernel.GENERIC<br />
cd kernel.GENERIC<br />
cd /usr/src<br />
make buildkernel installkernel<br />
<br />
make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null<br />
make installworld <br />
mergemaster -i<br />
</pre><br />
<br />
* populate /etc/rc.conf with IPs and NFS settings<br />
<pre>vi /etc/rc.conf<br />
<br />
hostname="nat2.johncompanies.com"<br />
kern_securelevel_enable="NO"<br />
portmap_enable="NO"<br />
sendmail_enable="NO"<br />
usbd_enable="YES"<br />
<br />
xntpd_enable="YES"<br />
xntpd_flags="-A -p /var/run/ntpd.pid"<br />
<br />
nfs_client_enable="YES"<br />
nfs_reserved_port_only="YES"<br />
ifconfig_em0="inet 10.1.6.50 netmask 255.255.255.0"<br />
#ifconfig_em0="inet 69.55.229.2 netmask 255.255.255.0"<br />
#ifconfig_em0_alias0="inet 69.55.229.229 netmask 255.255.255.255"<br />
ifconfig_fxp0="inet 69.55.229.2 netmask 255.255.255.0"<br />
ifconfig_fxp0_alias0="inet 69.55.229.3 netmask 255.255.255.255"<br />
ifconfig_fxp1="inet 10.1.2.1 netmask 255.255.255.0"<br />
defaultrouter="10.1.6.1"<br />
#defaultrouter=" 66.181.14.250"<br />
snmpd_enable="YES"<br />
ipnat_enable="YES"<br />
ipnat_rules="/etc/ipnat.rules"<br />
gateway_enable="YES"<br />
<br />
inetd_enable="YES"<br />
inetd_flags="-wW -a 10.1.2.1"<br />
fsck_y_enable="YES"<br />
background_fsck="NO"<br />
sshd_enable="YES"</pre><br />
<br />
* reboot. Confirm new kernel is loaded<br />
<br />
uname -a<br />
<br />
* update ports:<br />
<pre>cd /usr/ports<br />
echo "*default host=cvsup4.FreeBSD.org\<br />
*default base=/usr\<br />
*default prefix=/usr\<br />
*default release=cvs tag=RELENG_6_4\<br />
*default delete use-rel-suffix\<br />
*default compress\<br />
ports-all tag=." > sup<br />
<br />
cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null</pre><br />
<br />
* Install raid mgmt tool<br />
<br />
<pre>cd /usr/local/sbin<br />
fetch http://3ware.com/download/Escalade9690SA-Series/9.5.3/tw_cli-freebsd-x86-9.5.3.tgz<br />
tar xzf tw_cli-freebsd-x86-9.5.3.tgz<br />
rm tw_cli-freebsd-x86-9.5.3.tgz<br />
chmod 0700 tw_cli</pre><br />
<br />
Test:<br />
./tw_cli info c0<br />
<br />
* install rsync from ports<br />
<pre>cd /usr/ports/net/rsync<br />
make install clean</pre><br />
choose default options<br />
<br />
* install perl from ports<br />
<pre>cd /usr/ports/lang/perl5.8<br />
make install clean</pre><br />
<br />
* install screen from ports<br />
<pre>cd /usr/ports/sysutils/screen<br />
make install clean</pre><br />
<br />
* install bb client<br />
<pre>adduser<br />
Username: bb<br />
Full name: bb<br />
Uid (Leave empty for default): 1984<br />
Login group [bb]:<br />
Login group is bb. Invite bb into other groups? []:<br />
Login class [default]:<br />
Shell (sh csh tcsh nologin) [sh]: <br />
Home directory [/home/bb]:<br />
Use password-based authentication? [yes]:<br />
Use an empty password? (yes/no) [no]:<br />
Use a random password? (yes/no) [no]: yes<br />
Lock out the account after creation? [no]:<br />
Username : bb<br />
Password : <random><br />
Full Name : bb<br />
Uid : 1984<br />
Class :<br />
Groups : bb<br />
Home : /home/bb<br />
Shell : /bin/sh<br />
Locked : no<br />
OK? (yes/no): yes<br />
<br />
cd /usr/home/bb<br />
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .<br />
tar xvf bb-freebsd.tar</pre><br />
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:<br />
<pre>echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \<br />
10.1.2.1 nat2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts<br />
vi /home/bb/bbc1.9e-btf/ext/openfiles <br />
MACHINE="nat2,johncompanies,com" # HAS TO BE IN A,B,C FORM<br />
<br />
cd /usr/home/bb/bbc1.9e-btf/etc<br />
./bbchkcfg.sh <br />
(y to questions)<br />
./bbchkhosts.sh<br />
(ignore ssh errors)<br />
cd ../..<br />
chown -R bb .<br />
su bb<br />
cd<br />
cd bbc1.9e-btf/src<br />
make; make install<br />
cd ..<br />
<br />
vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh<br />
$1 $TOPARGS > $BBTMP/TOP.$$<br />
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$<br />
<br />
./runbb.sh start<br />
more BBOUT <br />
(look for errors)<br />
exit<br />
<br />
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh<br />
chmod +x /usr/local/etc/rc.d/bb.sh<br />
</pre><br />
<br />
Punch a hole in the firewall to allow it to communicate with bb monitor (probably already exists):<br />
<br />
ipfw add 96 allow ip from 66.181.18.0/27 to 69.55.230.2<br />
<br />
* configure bb on mail:<br />
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts<br />
64.163.14.48 nat2.johncompanies.com # ssh<br />
<br />
su bb<br />
cd<br />
bbsrc/bb/runbb.sh restart ; exit</pre><br />
<br />
* configure ntp<br />
<pre>echo "server 69.55.230.2<br />
server 0.pool.ntp.org<br />
server 1.pool.ntp.org<br />
server 2.pool.ntp.org<br />
server 3.pool.ntp.org<br />
server 66.187.233.4<br />
server 217.204.76.170<br />
server 64.112.189.11<br />
server 66.69.112.130<br />
server 80.85.129.25<br />
server 80.237.234.15<br />
server 130.60.7.44<br />
server 134.99.176.3<br />
server 198.144.202.250<br />
server 202.74.170.194<br />
server 204.17.42.199<br />
server 204.87.183.6<br />
server 213.15.3.1<br />
server 213.239.178.33<br />
server 217.114.97.97<br />
server 69.55.230.2" > /etc/ntp.conf</pre><br />
<br />
<pre>/usr/sbin/ntpd -A -p /var/run/ntpd.pid <br />
sleep 2; ntpq -p</pre><br />
(confirm it’s able to reach our time server)<br />
<br />
echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh<br />
chmod 0700 /usr/local/etc/rc.d/ntp.sh<br />
<br />
* fwd and reverse lookups on ns1c<br />
vr johncompanies.com<br />
(edit the PTR too)<br />
<br />
* setup backups, nfs mount<br />
<br />
<pre>mkdir /backup3<br />
echo 'backup3:/data /backup3 nfs rw,bg 0 0' >> /etc/fstab<br />
<br />
echo '#\!/bin/sh\<br />
backupdir=/data/nat2/current\<br />
\<br />
## ENTRY /etc ' > /usr/local/etc/backup.config</pre><br />
<br />
on backup3:<br />
setup backup dirs:<br />
ssh backup3 mkdir -p /data/nat2/current<br />
<br />
on backup3, add the system to <br />
vi /usr/local/sbin/snapshot_archive<br />
<br />
<pre>scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup <br />
vi /usr/local/etc/rsync.backup <br />
backup1 > backup3</pre><br />
<br />
crontab -e<br />
1 0 * * * /usr/local/etc/rsync.backup<br />
<br />
* edit sshd_config for security<br />
<pre>vi /etc/ssh/sshd_config<br />
ListenAddress 66.181.18.1<br />
ListenAddress 69.55.229.2<br />
ListenAddress 10.1.2.1<br />
<br />
kill -1 `cat /var/run/sshd.pid`</pre><br />
<br />
* raid chk<br />
<br />
<pre>cat > /usr/local/sbin/lsiraidchk<br />
#!/usr/bin/perl<br />
<br />
my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;<br />
<br />
foreach (@out) {<br />
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }<br />
#print $_;<br />
}</pre><br />
<br />
* netflow stuff<br />
add crontab entries<br />
<pre>crontab -e<br />
30 3 * * * /usr/local/etc/rsync.backup<br />
0 0 1 * * /sbin/ipfw zero<br />
0 0 1 * * /sbin/ipfw del 3 4 5<br />
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count<br />
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count<br />
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl; <br />
*/5 * * * * /usr/local/sbin/lsiraidchk<br />
<br />
#10 0 * * * rm /var/spool/clientmqueue/*</pre><br />
<br />
<pre>scp /etc/makefwrules.pl user@64.163.14.48:~<br />
scp /etc/makepiperules.pl user@64.163.14.48:~<br />
mv /home/user/makefwrules.pl /etc<br />
mv /home/user/makepiperules.pl /etc<br />
touch /etc/firewall.sh<br />
mkdir /etc/oldrules/</pre><br />
<br />
other binaries<br />
<br />
<pre>scp /usr/local/bin/rulemaker user@64.163.14.48:~<br />
mv ~user/rulemaker /usr/local/sbin<br />
scp ~user/Sendmail.pm user@64.163.14.48:~<br />
scp ~user/doswatch.pl user@64.163.14.48:~</pre><br />
<br />
* add nat rules<br />
<pre>vi /etc/ipnat.rules<br />
# sample entry<br />
bimap fxp0 10.1.6.70/32 -> 10.1.6.59/32<br />
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32<br />
<br />
ipnat -C -f /etc/ipnat.rules</pre><br />
<br />
* shell for user<br />
<pre>cp /root/.cshrc ~user/<br />
vi ~user/</pre><br />
change # to $<br />
<br />
* mrtg<br />
<br />
<pre>cd /usr/ports/net-mgmt/mrtg<br />
make install clean</pre><br />
(no FONTCONFIG, v3)<br />
<br />
this didn't work cause of libtool incompat<br />
<br />
so manually moved files:<br />
<br />
<pre>scp /usr/local/bin/cfgmaker user@nat2:/usr/local/bin/cfgmaker<br />
scp /usr/local/lib/perl5/site_perl/5.6.1/MRTG_lib.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/<br />
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_util.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/<br />
scp /usr/local/lib/perl5/site_perl/5.6.1/BER.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/<br />
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_Session.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/<br />
scp /usr/local/bin/mrtg root@nat2:/usr/local/bin/mrtg<br />
scp /usr/local/lib/perl5/site_perl/5.6.1/locales_mrtg.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/locales_mrtg.pm<br />
scp /usr/local/bin/rrdtool root@nat2:/usr/local/bin/rrdtool<br />
scp /usr/local/lib/perl5/site_perl/5.6.1/mach/RRDs.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/RRDs.pm<br />
rsync -av /usr/local/lib/perl5/site_perl/5.6.1/mach/auto/RRDs/ root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/auto/RRDs/<br />
scp /usr/lib/libz.so.2 root@nat2:/usr/lib/libz.so.2<br />
scp /usr/lib/libm.so.2 root@nat2:/usr/lib/libm.so.2<br />
rsync -av /usr/local/lib/librrd* root@nat2:/usr/local/lib/<br />
scp /usr/lib/libc.so.4 root@nat2:/usr/lib/libc.so.4<br />
<br />
rsync -av /usr/ports/net/rrdtool root@nat2:/usr/ports/net<br />
cd /usr/ports/net/rrdtool<br />
make install<br />
<br />
mkdir -p /mnt/data1/mrtg/data<br />
scp /usr/local/www/mgmt/mrtg/template.pl root@nat2:/mnt/data1/mrtg/<br />
scp /usr/local/www/mgmt/mrtg/host.pl root@nat2:/mnt/data1/mrtg/<br />
<br />
cfgmaker --if-template=template.pl --show-op-down --global "options[_]: growright,bits" --global 'WorkDir: /mnt/data1/mrtg/data' --global 'Interval: 1' --global 'LogFormat: rrdtool' --global 'PathAdd: /usr/local/bin' --global 'LibAdd: /usr/local/lib' --host-template=host.pl jc292401@10.1.2.50 --output=switch-p20.cfg<br />
<br />
cat > /mnt/data1/mrtg/mrtg.sh<br />
#!/bin/sh<br />
/usr/local/bin/mrtg /mnt/data1/mrtg/switch-p20.cfg<br />
<br />
chmod 0700 /mnt/data1/mrtg/mrtg.sh<br />
<br />
crontab -e<br />
* * * * * /mnt/data1/mrtg/mrtg.sh 2>&1 > /dev/null</pre><br />
<br />
* snmp firewall block<br />
<pre>cat > /usr/local/etc/rc.d/boot.sh<br />
ipfw add 10 allow udp from 69.55.230.2 to any 161<br />
ipfw add 10 allow udp from 10.1.2.1 to any 161<br />
ipfw add 11 deny udp from any to any 161<br />
chmod 0700 /usr/local/etc/rc.d/boot.sh</pre><br />
<br />
= bwdb =<br />
== Summary ==<br />
This machine tracks and stores network traffic (netflow) at castle. It is our means to monitor customer bandwidth usage.<br />
<br />
* Location: castle, cab 3-7<br />
* OS: FreeBSD 4.10 x86<br />
* Networking: Priv IP: 10.1.4.203 There are 2 onboard nic's, one of which is the "listener"<br />
* Hardware: Custom 1U. Single power supply. <br />
* Drives: one 250 GB (2 x 250GB) RAID1 array running on a Promise IDE RAID card.<br />
<br />
== Services Provided ==<br />
* netflow<br />
* mysql<br />
* bigbrother <br />
* snmp<br />
<br />
== netflow ==<br />
<br />
The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.<br />
<br />
A cronjob moves that flow file (or files if there are multiple due to some delay) <br />
1,16,31,46 * * * * /usr/home/flowbin/queue.pl<br />
<br />
into a processing queue:<br />
<tt>/usr/home/working</tt><br />
<br />
Then a separate file processes whatever flow files it finds there, inserting the data into the local mysql database:<br />
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl<br />
<br />
== mysql ==<br />
<br />
The database storing all the traffic data is named <tt>traffic</tt><br />
Tables:<br />
<pre>mysql> show tables;<br />
+---------------------------+<br />
| Tables_in_traffic |<br />
+---------------------------+<br />
| dailyIpTotals_69_55_224 |<br />
| dailyIpTotals_69_55_225 |<br />
| dailyIpTotals_69_55_226 |<br />
| dailyIpTotals_69_55_227 |<br />
| dailyIpTotals_69_55_228 |<br />
| dailyIpTotals_69_55_229 |<br />
| dailyIpTotals_69_55_230 |<br />
| dailyIpTotals_69_55_231 |<br />
| dailyIpTotals_69_55_232 |<br />
| dailyIpTotals_69_55_233 |<br />
| dailyIpTotals_69_55_234 |<br />
| dailyIpTotals_69_55_235 |<br />
| dailyIpTotals_69_55_236 |<br />
| dailyIpTotals_69_55_237 |<br />
| dailyIpTotals_69_55_238 |<br />
| dailyIpTotals_69_55_239 |<br />
| dailyPortTotals_69_55_224 |<br />
| dailyPortTotals_69_55_225 |<br />
| dailyPortTotals_69_55_226 |<br />
| dailyPortTotals_69_55_227 |<br />
| dailyPortTotals_69_55_228 |<br />
| dailyPortTotals_69_55_229 |<br />
| dailyPortTotals_69_55_230 |<br />
| dailyPortTotals_69_55_231 |<br />
| dailyPortTotals_69_55_232 |<br />
| dailyPortTotals_69_55_233 |<br />
| dailyPortTotals_69_55_234 |<br />
| dailyPortTotals_69_55_235 |<br />
| dailyPortTotals_69_55_236 |<br />
| dailyPortTotals_69_55_237 |<br />
| dailyPortTotals_69_55_238 |<br />
| dailyPortTotals_69_55_239 |<br />
| ipTotals_69_55_224 |<br />
| ipTotals_69_55_225 |<br />
| ipTotals_69_55_226 |<br />
| ipTotals_69_55_227 |<br />
| ipTotals_69_55_228 |<br />
| ipTotals_69_55_229 |<br />
| ipTotals_69_55_230 |<br />
| ipTotals_69_55_231 |<br />
| ipTotals_69_55_232 |<br />
| ipTotals_69_55_233 |<br />
| ipTotals_69_55_234 |<br />
| ipTotals_69_55_235 |<br />
| ipTotals_69_55_236 |<br />
| ipTotals_69_55_237 |<br />
| ipTotals_69_55_238 |<br />
| ipTotals_69_55_239 |<br />
| portTotals_69_55_224 |<br />
| portTotals_69_55_225 |<br />
| portTotals_69_55_226 |<br />
| portTotals_69_55_227 |<br />
| portTotals_69_55_228 |<br />
| portTotals_69_55_229 |<br />
| portTotals_69_55_230 |<br />
| portTotals_69_55_231 |<br />
| portTotals_69_55_232 |<br />
| portTotals_69_55_233 |<br />
| portTotals_69_55_234 |<br />
| portTotals_69_55_235 |<br />
| portTotals_69_55_236 |<br />
| portTotals_69_55_237 |<br />
| portTotals_69_55_238 |<br />
| portTotals_69_55_239 |<br />
+---------------------------+<br />
</pre><br />
<br />
So as you see we store each class-C block in its own table, for efficiency. Further, we store and organize data in 4 ways: "daily" tables and 15-minute granularity tables, and for each of those we track simple IP traffic and port-specific traffic. The daily tables contains 2 entries (one for each direction) for each IP for each day. For the current day, the row data is incremented as the day goes on. <br />
<br />
<pre>mysql> describe dailyIpTotals_69_55_224;<br />
+-----------+-------------+------+-----+---------+-------+<br />
| Field | Type | Null | Key | Default | Extra |<br />
+-----------+-------------+------+-----+---------+-------+<br />
| id | varchar(23) | | PRI | | |<br />
| date | date | YES | | NULL | |<br />
| ip | varchar(15) | YES | MUL | NULL | |<br />
| direction | tinyint(1) | YES | | NULL | |<br />
| octets | bigint(12) | YES | | NULL | |<br />
| packets | int(11) | YES | | NULL | |<br />
+-----------+-------------+------+-----+---------+-------+<br />
<br />
mysql> select * from dailyIpTotals_69_55_224 limit 1\G<br />
*************************** 1. row ***************************<br />
id: 6955224194-20100917-1<br />
date: 2010-09-17<br />
ip: 69.55.224.194<br />
direction: 1<br />
octets: 8821<br />
packets: 91<br />
</pre><br />
<br />
The <tt>id</tt> is a unique identifier (key), <tt>direction</tt> indicates incoming or outgoing traffic (outbound = 2, inbound = 1), <tt>octets</tt> are the amount of traffic in kilobytes, and <tt>packets</tt> is the total number of packets.<br />
<br />
The 15-minute table has similar information, but it's organized in 15 minute increments:<br />
<br />
<pre>mysql> describe ipTotals_69_55_224;<br />
+-----------+------------+------+-----+---------+-------+<br />
| Field | Type | Null | Key | Default | Extra |<br />
+-----------+------------+------+-----+---------+-------+<br />
| date | datetime | YES | | NULL | |<br />
| ip | char(15) | YES | MUL | NULL | |<br />
| direction | tinyint(1) | YES | | NULL | |<br />
| octets | bigint(20) | YES | | NULL | |<br />
| packets | int(11) | YES | | NULL | |<br />
+-----------+------------+------+-----+---------+-------+<br />
<br />
mysql> select * from ipTotals_69_55_224 limit 2\G<br />
*************************** 1. row ***************************<br />
date: 2010-01-11 19:30:00<br />
ip: 69.55.224.13<br />
direction: 1<br />
octets: 288<br />
packets: 6<br />
*************************** 2. row ***************************<br />
date: 2010-01-11 19:30:00<br />
ip: 69.55.224.12<br />
direction: 1<br />
octets: 216<br />
packets: 4</pre><br />
<br />
So for a given IP, there will be 192 rows in a given day: 4 rows per hour, *2 for 2 directions, *24 for 24hours in a day. Obviously this table is large which is why we broke it down into a daily table for quick, easy, daily-summary access.<br />
<br />
That covers the simple traffic tabulation tables. We also track traffic by port:<br />
<br />
<pre>mysql> describe dailyPortTotals_69_55_224;<br />
+-----------+-------------+------+-----+---------+-------+<br />
| Field | Type | Null | Key | Default | Extra |<br />
+-----------+-------------+------+-----+---------+-------+<br />
| id | varchar(28) | | PRI | | |<br />
| date | date | YES | | NULL | |<br />
| ip | varchar(15) | YES | MUL | NULL | |<br />
| direction | tinyint(1) | YES | | NULL | |<br />
| protocol | smallint(3) | YES | | NULL | |<br />
| port | int(11) | YES | | NULL | |<br />
| octets | bigint(11) | YES | | NULL | |<br />
| packets | int(11) | YES | | NULL | |<br />
+-----------+-------------+------+-----+---------+-------+<br />
8 rows in set (0.00 sec)<br />
<br />
mysql> select * from dailyPortTotals_69_55_224 limit 1\G<br />
*************************** 1. row ***************************<br />
id: 695522496-20091218-1-6-23<br />
date: 2009-12-18<br />
ip: 69.55.224.96<br />
direction: 1<br />
protocol: 6<br />
port: 23<br />
octets: 1796<br />
packets: 30<br />
<br />
mysql> select * from portTotals_69_55_224 limit 1\G<br />
*************************** 1. row ***************************<br />
date: 2010-09-07 18:45:00<br />
ip: 69.55.224.254<br />
direction: 1<br />
protocol: 6<br />
port: 99999<br />
octets: 144<br />
packets: 3<br />
<br />
</pre><br />
<br />
This is largely the same with 2 more additions: <tt>protocol</tt> (1=ICMP, 6=TCP, 17=UDP), and <tt>port</tt> which we set to 99999 if the traffic is return traffic and the port is above 1024. Obviously the potential for number of rows grows quickly when you consider the addition of port and protocol tracking per IP.<br />
<br />
== Regular maintenance ==<br />
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]<br />
* archive data from database<br />
archive_daily.pl 2012 09<br />
This will archive data for the given year and month from the daily summary tables. Generally we want to have a year of history in the database.<br />
<br />
archive_15min.pl 2012 09<br />
This will archive data for the given year and month from the 15min-increment tables. Generally, we want to have 6 months of history in the database.<br />
<br />
* if space becomes tight, move flow files and exported data to a backup server, both located in <tt>/usr/home/flowbin/archive</tt> and <tt>/usr/home/exported</tt>, respectively<br />
<br />
== Slaving ==<br />
<br />
If we were going to setup traffic database slaving (we don't do this anymore), perhaps cause the bwdb machine gets busy and it cannot handle traffic requests and netflow, here's how it's done:<br />
<br />
On the traffic master:<br />
<br />
GRANT REPLICATION SLAVE ON *.* TO 'repl'@'10.1.4.8' IDENTIFIED BY 'qERUG8wf';<br />
<br />
in my.cnf:<br />
<pre>bin-log<br />
server-id=1<br />
max_binlog_size=500M<br />
expire_logs_days = 3</pre><br />
<br />
on slave:<br />
in my.cnf:<br />
<pre>server-id = 2<br />
master-host = 10.1.4.203<br />
master-user = repl<br />
master-password = qERUG8wf<br />
master-connect-retry=60<br />
replicate-wild-do-table=traffic.daily%<br />
max_relay_log_size=500M<br />
expire_logs_days = 3<br />
<br />
replicate-wild-do-table=traffic.%</pre><br />
<br />
on master:<br />
touch /usr/home/working/.lock<br />
(make sure processflows not running)<br />
<br />
<pre>FLUSH TABLES WITH READ LOCK;<br />
cd /usr/home/database/traffic<br />
tar -czf mysql-traffic-snapshot.tgz ./daily*<br />
(~1G)<br />
SHOW MASTER STATUS;<br />
+-----------------+-----------+--------------+------------------+<br />
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |<br />
+-----------------+-----------+--------------+------------------+<br />
| bwdb-bin.000039 | 154432615 | | |<br />
+-----------------+-----------+--------------+------------------+<br />
<br />
(write down info)<br />
UNLOCK TABLES;<br />
scp mysql-traffic-snapshot.tgz 10.1.4.5:/mnt/data1/db/mysql/traffic/</pre><br />
<br />
on slave:<br />
<pre>mkdir /mnt/data1/db/mysql/traffic<br />
cd /mnt/data1/db/mysql/traffic/<br />
tar xzvf mysql-traffic-snapshot.tgz<br />
(restart mysql)<br />
CHANGE MASTER TO MASTER_HOST='10.1.4.203',MASTER_USER='repl',MASTER_PASSWORD='qERUG8wf',MASTER_LOG_FILE='bwdb-bin.000059',MASTER_LOG_POS=482502186;<br />
START SLAVE;</pre><br />
<br />
<pre>cd /usr/home/database/traffic<br />
scp *</pre> <br />
<br />
<pre>optimize table dailyPortTotals_69_55_224;<br />
optimize table dailyPortTotals_69_55_225;<br />
optimize table dailyPortTotals_69_55_226;<br />
optimize table dailyPortTotals_69_55_227;<br />
optimize table dailyPortTotals_69_55_228;<br />
optimize table dailyPortTotals_69_55_229;<br />
optimize table dailyPortTotals_69_55_230;<br />
optimize table dailyPortTotals_69_55_231;<br />
optimize table dailyPortTotals_69_55_232;<br />
optimize table dailyPortTotals_69_55_233;<br />
optimize table dailyPortTotals_69_55_234;<br />
optimize table dailyPortTotals_69_55_235;<br />
optimize table dailyPortTotals_69_55_236;<br />
optimize table dailyPortTotals_69_55_237;<br />
optimize table dailyPortTotals_69_55_238;<br />
optimize table dailyPortTotals_69_55_239;</pre><br />
<br />
== Build ==<br />
<br />
=== BIOS Config ===<br />
disable quiet boot<br />
<br />
set to last state after power loss<br />
<br />
set date/time to GMT<br />
<br />
enable serial console output (baud rate 115200)<br />
<br />
=== Install OS ===<br />
<br />
Install FreeBSD 8.3 amd64<br />
<br />
* partition map:<br />
<pre>/ 500m<br />
swap 4096m<br />
/var 256m<br />
/tmp 256m<br />
/usr ~</pre><br />
<br />
* edit /etc/make.conf <br />
Castle:<br />
<pre>echo "WITHOUT_X11=yes \<br />
KERNCONF=bwdb \<br />
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre><br />
<br />
i2b:<br />
<pre>echo "WITHOUT_X11=yes \<br />
KERNCONF=bwdb2 \<br />
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre><br />
<br />
* add settings to /boot/loader.conf and /boot.config<br />
<br />
<pre>echo "-Dh" >> /boot.config<br />
<br />
echo 'console="comconsole,vidconsole" \<br />
boot_multicons="YES" \<br />
boot_serial="YES" \<br />
comconsole_speed="115200"' >> /boot/loader.conf</pre><br />
<br />
* turn off all ttyv's except 0 and 1 in /etc/ttys<br />
also turn on ttyu0, change type to vt100:<br />
<pre>vi /etc/ttys<br />
<br />
ttyv2 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv3 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv4 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv5 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv6 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv7 "/usr/libexec/getty Pc" cons25 off secure<br />
# Serial terminals<br />
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.<br />
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure<br />
<br />
kill -1 1</pre><br />
<br />
on console server:<br />
vi /etc/remote<br />
(rename port to jail8 depending on where and which digi plugged into)<br />
test serial console<br />
<br />
<br />
* populate hosts<br />
i2b:<br />
<pre>echo "69.55.230.10 backup2" >> /etc/hosts<br />
echo "69.55.230.11 backup1" >> /etc/hosts<br />
echo "10.1.2.3 backup3" >> /etc/hosts</pre><br />
<br />
castle:<br />
<pre>echo "10.1.4.3 backup2 backup2.johncompanies.com" >> /etc/hosts<br />
echo "10.1.4.8 backup1 backup1.johncompanies.com" >> /etc/hosts<br />
echo "10.1.4.4 mail mail.johncompanies.com" >> /etc/hosts<br />
</pre><br />
<br />
* put key in authorized_keys on backup1 and backup2<br />
cd<br />
ssh-keygen -t dsa -b 1024 <br />
(default location, leave password blank)<br />
<br />
castle:<br />
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys' <br />
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' <br />
<br />
i2b:<br />
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys' <br />
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' <br />
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys' <br />
<br />
confirm that you can ssh to backup2 and backup1 (and backup3 if at i2b) without getting a login prompt<br />
<br />
ssh backup1 hostname<br />
ssh backup2 hostname<br />
<br />
* edit root's path and login script:<br />
vi /root/.cshrc<br />
<br />
Change alias entries (add G):<br />
<pre>alias la ls -aG<br />
alias lf ls -FAG<br />
alias ll ls -lAG<br />
alias ls ls -AG<br />
</pre><br />
<br />
and alter the prompt, set the following:<br />
set prompt = "`/bin/hostname -s` %/# "<br />
<br />
* install cvsup<br />
cd /usr/ports/net/cvsup-without-gui <br />
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null<br />
<br />
* get latest sources for this release:<br />
<pre>cd /usr/src <br />
echo "*default host=cvsup4.freebsd.org\<br />
*default base=/usr\<br />
*default prefix=/usr\<br />
*default release=cvs tag=RELENG_8_3\<br />
*default delete use-rel-suffix\<br />
*default compress\<br />
src-all" > sup<br />
<br />
cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null</pre><br />
<br />
* configure new kernel<br />
<br />
cd /usr/src/sys/amd64/conf <br />
scp backup2:/mnt/data4/build/freebsd/kern_config-bwdb-8.3-amd64 ./bwdb<br />
<br />
Edit config and change name:<br />
vi bwdb<br />
ident bwdb<br />
<br />
* build, install kernel and world<br />
<br />
<pre>cd /boot<br />
<br />
mv kernel kernel.GENERIC<br />
cd kernel.GENERIC<br />
cd /usr/src<br />
make buildkernel installkernel<br />
<br />
make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null<br />
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)<br />
make installworld <br />
(2450: 3min, supermicro: 1min, 2950: :34)<br />
mergemaster -i</pre><br />
<br />
* populate /etc/rc.conf with IPs and NFS settings<br />
<br />
castle:<br />
<pre>vi /etc/rc.conf<br />
<br />
hostname="bwdb.johncompanies.com"<br />
kern_securelevel_enable="NO"<br />
portmap_enable="NO"<br />
sendmail_enable="NO"<br />
usbd_enable="YES"<br />
<br />
xntpd_enable="YES"<br />
nfs_client_enable="YES"<br />
nfs_reserved_port_only="YES"<br />
ifconfig_fxp0="inet 10.1.4.203 netmask 255.255.255.0"<br />
ifconfig_em0="up promisc"<br />
defaultrouter="10.1.4.1"<br />
snmpd_enable="YES"<br />
<br />
inetd_enable="YES"<br />
inetd_flags="-wW -a 10.1.4.203"<br />
fsck_y_enable="YES"<br />
background_fsck="NO"<br />
sshd_enable="YES"<br />
ipfw_load="YES"</pre><br />
<br />
i2b:<br />
<pre>vi /etc/rc.conf<br />
<br />
hostname="bwdb2.johncompanies.com"<br />
kern_securelevel_enable="NO"<br />
portmap_enable="NO"<br />
sendmail_enable="NO"<br />
usbd_enable="YES"<br />
<br />
xntpd_enable="YES"<br />
nfs_client_enable="YES"<br />
nfs_reserved_port_only="YES"<br />
ifconfig_fxp0="inet 10.1.2.4 netmask 255.255.255.0"<br />
ifconfig_em0="up promisc"<br />
defaultrouter="10.1.2.1"<br />
snmpd_enable="YES"<br />
<br />
inetd_enable="YES"<br />
inetd_flags="-wW -a 10.1.2.4"<br />
fsck_y_enable="YES"<br />
background_fsck="NO"<br />
sshd_enable="YES"<br />
ipfw_load="YES"</pre><br />
<br />
* reboot. Confirm new kernel is loaded<br />
<br />
uname -a<br />
<br />
* update ports:<br />
<pre>cd /usr/ports<br />
echo "*default host=cvsup4.FreeBSD.org\<br />
*default base=/usr\<br />
*default prefix=/usr\<br />
*default release=cvs tag=RELENG_8_3\<br />
*default delete use-rel-suffix\<br />
*default compress\<br />
ports-all tag=." > sup<br />
<br />
cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre><br />
<br />
* Install raid mgmt tool<br />
<br />
<pre>cd /usr/local/sbin<br />
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz .<br />
tar xzf tw_cli-freebsd-x86_64-9.5.0.1.tgz<br />
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz<br />
chmod 0700 tw_cli</pre><br />
<br />
Test:<br />
./tw_cli info c0<br />
<br />
Grab raid check script:<br />
scp backup1:/usr/local/sbin/3wraidchk /usr/local/etc<br />
<br />
Setup cronjob:<br />
<pre>crontab -e<br />
*/5 * * * * /usr/local/etc/3wraidchk</pre><br />
<br />
* install rsync from ports<br />
cd /usr/ports/net/rsync<br />
make install clean<br />
<br />
choose default options<br />
<br />
* install perl from ports<br />
cd /usr/ports/lang/perl5.8<br />
make install clean<br />
<br />
choose default options<br />
<br />
* install bb client<br />
<br />
Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux: <br />
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/<br />
<br />
adduser<br />
<br />
Output/response:<br />
<br />
<pre>Username: bb<br />
Full name: bb<br />
Uid (Leave empty for default): 1984<br />
Login group [bb]:<br />
Login group is bb. Invite bb into other groups? []:<br />
Login class [default]:<br />
Shell (sh csh tcsh nologin) [sh]: <br />
Home directory [/home/bb]:<br />
Use password-based authentication? [yes]:<br />
Use an empty password? (yes/no) [no]:<br />
Use a random password? (yes/no) [no]: yes<br />
Lock out the account after creation? [no]:<br />
Username : bb<br />
Password : <random><br />
Full Name : bb<br />
Uid : 1984<br />
Class :<br />
Groups : bb<br />
Home : /home/bb<br />
Shell : /bin/sh<br />
Locked : no<br />
OK? (yes/no): yes</pre><br />
<br />
cd /usr/home/bb<br />
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .<br />
tar xzf bb-freebsd_linuxcompat.tgz<br />
<br />
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:<br />
<br />
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \<br />
10.1.4.203 bwdb.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts<br />
<br />
Edit for machine name and private IP.<br />
<br />
if this machine is at i2b:<br />
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \<br />
10.1.2.4 bwdb2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts<br />
<br />
<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles <br />
<br />
MACHINE="bwdb,johncompanies,com" # HAS TO BE IN A,B,C FORM</pre><br />
Edit for machine name.<br />
<br />
Have bb watch for flow-capture, mysql<br />
<pre>cat >> /home/bb/bbc1.9e-btf/etc/bb-proctab<br />
localhost: flow-capture :<br />
localhost: mysqld :</pre><br />
<br />
<br />
<pre>cd /usr/home/bb/bbc1.9e-btf/etc<br />
./bbchkcfg.sh <br />
(y to questions)<br />
./bbchkhosts.sh<br />
(ignore ssh errors)<br />
cd ../..<br />
chown -R bb .<br />
su bb<br />
cd<br />
cd bbc1.9e-btf<br />
./runbb.sh start<br />
more BBOUT <br />
(look for errors)<br />
exit</pre><br />
<br />
Put in script to start bb @ boot:<br />
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh<br />
chmod +x /usr/local/etc/rc.d/bb.sh<br />
<br />
<br />
If this is at i2b, punch a hole in the firewall to allow it to communicate with bb monitor:<br />
<br />
ipfw add 00096 allow tcp from 66.181.18.0/27 to 69.55.230.2<br />
<br />
<br />
* configure bb on mail<br />
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts<br />
10.1.4.203 bwdb.johncompanies.com # ssh<br />
<br />
su bb<br />
cd<br />
bbsrc/bb/runbb.sh restart ; exit</pre><br />
<br />
* configure ntp server<br />
Castle:<br />
echo "server 10.1.4.1" > /etc/ntp.conf<br />
<br />
I2b:<br />
echo "server 10.1.2.1" > /etc/ntp.conf<br />
<br />
<pre>/usr/sbin/ntpd -p /var/run/ntpd.pid<br />
sleep 2; ntpq -p</pre><br />
(confirm it’s able to reach our time server)<br />
<br />
<pre>echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh<br />
chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre><br />
<br />
* fwd and reverse lookups on ns1c<br />
vr johncompanies.com<br />
(edit the PTR too)<br />
<br />
* setup backups<br />
<pre>echo '#\!/bin/sh\<br />
backupdir=/data/bwdb/current\<br />
server=backup1\<br />
\<br />
## ENTRY /etc\<br />
## ENTRY /usr/home/flowbin\<br />
## ENTRY /usr/home/database' > /usr/local/etc/backup.config</pre><br />
<br />
Castle:<br />
setup backup dirs:<br />
ssh backup1 mkdir -p /data/bwdb/current<br />
on backup1, add the system to <br />
vi /usr/local/sbin/snapshot_rotate<br />
<br />
I2b:<br />
setup backup dirs:<br />
ssh backup3 mkdir -p /data/bwdb/current<br />
on backup3, add the system to <br />
vi /usr/local/sbin/snapshot_archive<br />
<br />
<br />
Copy over the backup script:<br />
scp backup2:/d4/bin/freebsd8.x/rsync.backup /usr/local/etc/<br />
<br />
Edit rsync.backup and change <tt>config</tt> var to point to correct config file location: <tt>/usr/local/etc/backup.config</tt><br />
<br />
<pre>crontab -e<br />
5 0 * * * /usr/local/etc/rsync.backup</pre><br />
<br />
* make /root/logs<br />
mkdir /root/logs<br />
<br />
* edit sshd_config for security<br />
<br />
<pre>vi /etc/ssh/sshd_config<br />
ListenAddress 10.1.4.203<br />
PermitRootLogin yes<br />
<br />
kill -1 `cat /var/run/sshd.pid`</pre><br />
<br />
Edit for private IP.<br />
<br />
* snmp<br />
<br />
(Before doing this you may need to take down the firewall and also add to resolv.conf 69.43.143.41)<br />
<pre><br />
cd /usr/ports/net-mgmt/net-snmp<br />
make install clean<br />
(defaults)<br />
<br />
cat >> /etc/rc.conf<br />
snmpd_enable="YES"<br />
snmpd_flags="-a"<br />
snmpd_conffile="/usr/local/share/snmp/snmpd.conf"<br />
snmptrapd_enable="YES"<br />
snmptrapd_flags="-a -p /var/run/snmptrapd.pid"<br />
<br />
cat > /usr/local/share/snmp/snmpd.conf<br />
rocommunity jcread 10.1.4.5<br />
rocommunity jcread 10.1.4.202<br />
</pre><br />
<br />
=== netflow ===<br />
<br />
Install flow tools:<br />
<pre>cd /usr/ports/net-mgmt/flow-tools<br />
make install clean</pre><br />
Defaults.<br />
<br />
mkdir /usr/home/flows<br />
<br />
Flow start script:<br />
echo "/usr/local/bin/flow-capture -w /usr/home/flows -S5 -N -2 0/10.1.4.203/4444" > /usr/local/etc/rc.d/flow-capture.sh<br />
<br />
chmod 0700 /usr/local/etc/rc.d/flow-capture.sh<br />
<br />
Edit for private IP.<br />
<br />
Netgraph start script:<br />
<pre><br />
cat > /usr/local/etc/rc.d/netgraph.sh<br />
<br />
/usr/sbin/ngctl -f- <<-SEQ<br />
mkpeer em0: netflow lower iface0<br />
name em0:lower netflow<br />
connect em0: netflow: upper out0<br />
mkpeer netflow: ksocket export inet/dgram/udp<br />
msg netflow:export connect inet/10.1.4.203:4444<br />
SEQ<br />
<br />
#/usr/sbin/ngctl -f- <<-SEQ<br />
#shutdown netflow:<br />
#SEQ<br />
<br />
chmod 0700 /usr/local/etc/rc.d/netgraph.sh</pre><br />
Edit for private IP.<br />
<br />
Confirm netflow is running after running scripts:<br />
<pre>newbwdb /usr/ports/net-mgmt/flow-tools# /usr/sbin/ngctl<br />
Available commands:<br />
config get or set configuration of node at <path><br />
connect Connects hook <peerhook> of the node at <relpath> to <hook><br />
debug Get/set debugging verbosity level<br />
dot Produce a GraphViz (.dot) of the entire netgraph.<br />
help Show command summary or get more help on a specific command<br />
list Show information about all nodes<br />
mkpeer Create and connect a new node to the node at "path"<br />
msg Send a netgraph control message to the node at "path"<br />
name Assign name <name> to the node at <path><br />
read Read and execute commands from a file<br />
rmhook Disconnect hook "hook" of the node at "path"<br />
show Show information about the node at <path><br />
shutdown Shutdown the node at <path><br />
status Get human readable status information from the node at <path><br />
types Show information about all installed node types<br />
write Send a data packet down the hook named by "hook".<br />
quit Exit program<br />
+ show netflow:<br />
Name: netflow Type: netflow ID: 00000004 Num hooks: 3<br />
Local hook Peer name Peer type Peer ID Peer hook<br />
---------- --------- --------- ------- ---------<br />
export <unnamed> ksocket 00000005 inet/dgram/udp<br />
out0 em0 ether 00000001 upper<br />
iface0 em0 ether 00000001 lower<br />
+<br />
</pre><br />
<br />
We notice that sometimes flow-capture is failing due to swap exhaustion (even after adding more swap). So we crontab flow-capture to restart (it's ok to start if it's already running, it just quits):<br />
<br />
<pre><br />
crontab -e<br />
#restart flow-capture<br />
*/15 * * * * /usr/local/etc/rc.d/flow-capture.sh<br />
</pre><br />
<br />
==== process flow tools ==== <br />
<br />
<pre>mkdir /usr/home/flowbin<br />
mkdir /usr/home/working</pre><br />
<br />
Install modules:<br />
<pre>cd /usr/ports/devel/p5-Date-Calc<br />
make install clean<br />
cd /usr/ports/mail/p5-Mail-Sendmail<br />
make install clean</pre><br />
<br />
Queue script:<br />
<pre><br />
cat > /usr/home/flowbin/queue.pl<br />
#!/usr/bin/perl<br />
<br />
use strict;<br />
<br />
BEGIN {<br />
push @INC, "/usr/home/flowbin";<br />
}<br />
<br />
use date;<br />
<br />
my $flowbase = "/usr/home/flows";<br />
#my $flowqueue = "/usr/home/queue";<br />
my $flowqueue = "/usr/home/working";<br />
<br />
my ($date, $time) = date::CurrentDateTime();<br />
<br />
my $flowdir = mkFlowDir($date);<br />
`mv $flowdir/ft-* $flowqueue`;<br />
<br />
if (date::DateWindow($date, $time, $date, "00:00:00", 600)) {<br />
my $newdate = date::AddDays($date, -1);<br />
my $flowdir = mkFlowDir($newdate);<br />
`mv $flowdir/ft-* $flowqueue`;<br />
}<br />
<br />
sub mkFlowDir {<br />
my $date = shift;<br />
$date =~ /([0-9]{4}-[0-9]{2})/;<br />
my $yearmonth = $1;<br />
return "$flowbase/$yearmonth/$date";<br />
}<br />
</pre><br />
<br />
Date.pm module:<br />
<pre><br />
cat > /usr/home/flowbin/date.pm<br />
#!/usr/local/bin/perl<br />
#<br />
# $Header: /usr/cvs/newgw/lib/date.pm,v 1.2 2003/11/24 17:06:02 glenn Exp $<br />
#<br />
# Copyright (c) 2001, 2002, 2003<br />
# e-Monitoring Networks, Inc. All rights reserved.<br />
#<br />
#<br />
#<br />
# date.pl - Higher level functions written on top of Date::Calc<br />
<br />
package date;<br />
<br />
use strict;<br />
use Date::Calc qw(:all);<br />
<br />
sub DayDiff { #calculate the difference in days from two dates<br />
my $date1 = shift;<br />
my $date2 = shift;<br />
my ($year1, $month1, $day1) = &DateToymd($date1);<br />
my ($year2, $month2, $day2) = &DateToymd($date2);<br />
my $diff = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);<br />
return $diff;<br />
}<br />
<br />
sub AddDays { #adds specified number of days to the supplied date<br />
my $date = shift;<br />
my $days = shift;<br />
my ($year, $month, $day) = &DateToymd($date);<br />
my ($nyear, $nmonth, $nday) = &Add_Delta_Days($year, $month, $day, $days);<br />
my $ndate = &ymdToDate($nyear, $nmonth, $nday);<br />
return $ndate;<br />
}<br />
<br />
sub AddHours { #adds specified number of hours to the supplied date and time<br />
my $date = shift;<br />
my $time = shift;<br />
my $addhours = shift;<br />
my $adddays = 0;<br />
if (abs($addhours / 24) >= 1) {<br />
$adddays = int($addhours / 24);<br />
$addhours -= $adddays * 24;<br />
}<br />
my ($year, $month, $day) = &DateToymd($date);<br />
my ($hour, $minute, $second) = &TimeTohms($time);<br />
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,<br />
$hour, $minute, $second,<br />
$adddays, $addhours, 0, 0);<br />
my $ndate = &ymdToDate($ny, $nm, $nd);<br />
my $ntime = &hmsToTime($nh, $nmin, $ns);<br />
return $ndate, $ntime;<br />
}<br />
<br />
sub AddMinutes {<br />
my $date = shift;<br />
my $time = shift;<br />
my $minutes = shift;<br />
my ($year, $month, $day) = &DateToymd($date);<br />
my ($hour, $minute, $second) = &TimeTohms($time);<br />
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,<br />
$hour, $minute, $second,<br />
0, 0, $minutes, 0);<br />
my $ndate = &ymdToDate($ny, $nm, $nd);<br />
my $ntime = &hmsToTime($nh, $nmin, $ns);<br />
return $ndate, $ntime;<br />
}<br />
<br />
sub CurrentDateTime { #return the current date and time<br />
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;<br />
my $date = &ymdToDate($y, $m, $d);<br />
my $time = &hmsToTime($h, $min, $s);<br />
return $date, $time;<br />
}<br />
<br />
sub Currentymd { #return the current year, month and day as separate variables<br />
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;<br />
return $y, $m, $d;<br />
}<br />
<br />
sub DateToymd { #takes a date and returns year, month, day as individual values<br />
my $date = shift;<br />
if ($date =~ /([0-9]{4})-([0-9]{2})-([0-9]{2})/) {<br />
my $day = $3;<br />
my $month = $2;<br />
my $year = $1;<br />
return $year, $month, $day;<br />
}<br />
return undef;<br />
}<br />
<br />
sub TimeTohms { #takes a time and return hours minutes and seconds as individual values<br />
my $time = shift;<br />
if ($time =~ /([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})/) {<br />
my $hour = $1;<br />
my $minute = $2;<br />
my $second = $3;<br />
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }<br />
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }<br />
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }<br />
return $hour, $minute, $second;<br />
}<br />
return undef;<br />
}<br />
<br />
sub ymdToDate { #takes year, month, day and assembles them into our date format<br />
my $year = shift;<br />
my $month = shift;<br />
my $day = shift;<br />
if (defined($year) && defined($month) && defined ($day)) {<br />
$month = sprintf("%02d", $month);<br />
$day = sprintf("%02d", $day);<br />
return "$year-$month-$day";<br />
}<br />
return undef;<br />
}<br />
<br />
sub hmsToTime { #takes hour minute and second and assembles them into our time format<br />
my $hour = shift;<br />
my $minute = shift;<br />
my $second = shift;<br />
if (defined($hour) && defined($minute) && defined ($second)) {<br />
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }<br />
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }<br />
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }<br />
return sprintf ("%02d:%02d:%02d", $hour, $minute, $second);<br />
}<br />
return undef;<br />
}<br />
<br />
sub CompareDates { #compares two date and time pairs<br />
my $date1 = shift;<br />
my $time1 = shift;<br />
my $date2 = shift;<br />
my $time2 = shift;<br />
<br />
my ($year1, $month1, $day1) = &DateToymd($date1);<br />
my ($hour1, $minute1, $second1) = &TimeTohms($time1);<br />
my ($year2, $month2, $day2) = &DateToymd($date2);<br />
my ($hour2, $minute2, $second2) = &TimeTohms($time2);<br />
<br />
# &debug("$year1, $month1, $day1, $year2, $month2, $day2");<br />
my $days = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);<br />
if ($days > 0) { return 1;}<br />
if ($days < 0) { return -1;}<br />
if ($days == 0) { #same day, compare times<br />
my $seconds1 = $second1 + (60 * $minute1) + (3600 * $hour1);<br />
my $seconds2 = $second2 + (60 * $minute2) + (3600 * $hour2);<br />
if ($seconds1 < $seconds2) { return 1;}<br />
if ($seconds1 > $seconds2) { return -1;}<br />
if ($seconds1 == $seconds2) { return 0;}<br />
}<br />
return undef;<br />
}<br />
<br />
sub DateWindow { #compares two date time pairs to see if they are < X seconds apart<br />
my $date1 = shift;<br />
my $time1 = shift;<br />
my $date2 = shift;<br />
my $time2 = shift;<br />
my $window = shift;<br />
<br />
my ($year1, $month1, $day1) = &DateToymd($date1);<br />
my ($hour1, $minute1, $second1) = &TimeTohms($time1);<br />
my ($year2, $month2, $day2) = &DateToymd($date2);<br />
my ($hour2, $minute2, $second2) = &TimeTohms($time2);<br />
<br />
my ($day, $hour, $minute, $second) =<br />
&Delta_DHMS($year1, $month1, $day1, $hour1, $minute1, $second1,<br />
$year2, $month2, $day2, $hour2, $minute2, $second2);<br />
$minute *= 60;<br />
$hour *= 3600;<br />
$day *= 86400;<br />
my $total = $second + $minute + $hour + $day;<br />
if (abs($total) < $window) {<br />
return 1;<br />
}<br />
return 0;<br />
}<br />
<br />
sub CheckDateOrder { #takes three dates/times, returns true if they are in chronological order<br />
my $date1 = shift;<br />
my $time1 = shift;<br />
my $date2 = shift;<br />
my $time2 = shift;<br />
my $date3 = shift;<br />
my $time3 = shift;<br />
if (&CompareDates($date1, $time1, $date2, $time2) == -1) {<br />
return 0;<br />
}<br />
if (&CompareDates($date2, $time2, $date3, $time3) == -1) {<br />
return 0;<br />
}<br />
return 1;<br />
}<br />
<br />
sub EpochSeconds { #calculates number of seconds since the epoch for the given date/time<br />
my $date = shift;<br />
my $time = shift;<br />
my ($year, $month, $day) = &DateToymd($date);<br />
my ($hour, $minute, $second) = &TimeTohms($time);<br />
my ($d, $h, $m, $s) = &Delta_DHMS(1970, 1, 1, 0, 0, 0,<br />
$year, $month, $day, $hour, $minute, $second);<br />
my $seconds = $s + (60 * $m) + (3600 * $h) + (86400 * $d);<br />
return $seconds;<br />
}<br />
<br />
sub SecondsToDateTime { #converts seconds since epoch to date/time<br />
my $seconds = shift;<br />
my $days = int($seconds / 86400);<br />
$seconds -= $days * 86400;<br />
my $hours = int($seconds / 3600);<br />
$seconds -= $hours * 3600;<br />
my $minutes = int($seconds / 60);<br />
$seconds -= $minutes * 60;<br />
my ($year, $month, $day, $hour, $minute, $second) =<br />
&Add_Delta_DHMS(1970, 1, 1, 0, 0, 0, $days, $hours, $minutes, $seconds);<br />
$month = sprintf("%02d", $month);<br />
$day = sprintf("%02d", $day);<br />
$hour = sprintf("%02d", $hour);<br />
$minute = sprintf("%02d", $minute);<br />
$second = sprintf("%02d", $second);<br />
return "$year-$month-$day", "$hour:$minute:$second";<br />
}<br />
<br />
sub DateToDayName {<br />
my $date = shift;<br />
my ($year, $month, $day) = &DateToymd($date);<br />
my $name = &Day_of_Week_to_Text(&Day_of_Week($year, $month, $day));<br />
$name =~ /^[A-Za-z]{3}/;<br />
$name = $&;<br />
return $name;<br />
}<br />
<br />
sub ValiDate {<br />
return @_;<br />
<br />
<br />
<br />
}<br />
<br />
sub CheckBusinessDay { # checks to see if date is business day. 1=yes, 0=no<br />
my $date = shift;<br />
my ($year, $month, $day) = &DateToymd($date);<br />
if (Day_of_Week($year,$month,$day) < 6) { return 1; }<br />
else { return 0; }<br />
}<br />
<br />
1; #don't remove this line<br />
</pre><br />
<br />
chmod 0700 /usr/home/flowbin/queue.pl<br />
<br />
Setup cronjob:<br />
<pre>crontab -e<br />
#move flow data into the queue<br />
1,16,31,46 * * * * /usr/home/flowbin/queue.pl</pre><br />
<br />
==== flow processing: i2b ==== <br />
<pre>cat > /usr/home/flowbin/processflows-sql.pl<br />
#!/usr/bin/perl<br />
<br />
#use strict;<br />
#$debug=1;<br />
#$dry=1;<br />
<br />
my $log = '/usr/home/flowbin/discards.log';<br />
<br />
use Data::Dumper;<br />
<br />
BEGIN {<br />
push @INC, "/usr/home/flowbin";<br />
}<br />
<br />
#my $queuedir = "/usr/home/queue";<br />
my $queuedir = "/usr/home/working";<br />
my $archivedir = "/usr/home/archive";<br />
my $sqldir = "/usr/home/sql";<br />
my $sqldirworking = "/usr/home/sql/tmp";<br />
<br />
unless ($dry) {<br />
if (-e "$queuedir/.lock") {<br />
open(FILE, "$queuedir/.lock");<br />
my $pid = <FILE>;<br />
chomp($pid);<br />
close(FILE);<br />
if (kill(0, $pid)) {<br />
#another process is using the queue, bail out<br />
exit(0);<br />
}<br />
else {<br />
#dead lock file, remove it<br />
`rm $queuedir/.lock`;<br />
}<br />
}<br />
open(FILE, "> $queuedir/.lock");<br />
print FILE "$$\n";<br />
close(FILE);<br />
}<br />
<br />
opendir(DIR, $queuedir);<br />
my @files = readdir(DIR);<br />
closedir(DIR);<br />
<br />
foreach my $file (sort @files) {<br />
unless($file =~ /^\./) {<br />
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;<br />
my $date = "$1 $2:$3:$4";<br />
my $outfile = "$1-$2:$3.sql";<br />
unless (open (SQL, "+> $sqldirworking/$outfile")) { die "cant open $sqldirworking/$outfile"; }<br />
my $condensedDate = $1;<br />
$condensedDate =~ s/-//g;<br />
my $iptotal = {};<br />
my $protototal = {};<br />
my $porttotal = {};<br />
<br />
&debug("started file $file at ");<br />
&debug(`date`);<br />
&debug("getting raw flow data (flow-print)");<br />
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;<br />
&debug("aggregating data at ");<br />
&debug(`date`);<br />
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }<br />
LOOP: while (my $line = readline DATA) {<br />
my @d = split /[\s]+/, $line;<br />
if ($d[0] ne '' && $d[0] ne 'Start') {<br />
my $addr = 0;<br />
my $port = 0;<br />
<br />
#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets<br />
#0 1 2 3 4 5 6 7 8 9 10 11<br />
#|<br />
# outbound = 2, inbound = 1<br />
<br />
my (@src_ip) = split '\.', $d[3];<br />
my (@dst_ip) = split '\.', $d[6];<br />
<br />
if ($src_ip[0] == 69 && $src_ip[1] == 55 && ($src_ip[2] == 229 || $src_ip[2] == 231)) { # for i2b<br />
$d[2] = 2;<br />
# hack for outbound bulk traffic counted 2x<br />
#if ($src_ip[2] == 231) { $d[11] /= 2; $d[10] /= 2; }<br />
}<br />
# note- this is where we filter out IPs only found at i2b<br />
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && ($dst_ip[2] == 229 || $dst_ip[2] == 231)) { # for i2b<br />
$d[2] = 1;<br />
}<br />
else { next LOOP; }<br />
<br />
if ($d[2] == 2) {<br />
$addr = $d[3];<br />
# if the dst-port is low, store that<br />
if ($d[7] <= 1024) { $port = $d[7]; }<br />
# if the src-port is low, store that<br />
elsif ($d[4] <= 1024) { $port = $d[4]; }<br />
else { $port = 99999; }<br />
}<br />
elsif ($d[2] == 1) {<br />
$addr = $d[6];<br />
# if the dst-port is high, assume its return traffic, try to store src-port if low<br />
if ($d[7] > 1024) {<br />
if ($d[4] <= 1024) { $port = $d[4]; }<br />
else { $port = 99999; }<br />
} else {<br />
$port = $d[7];<br />
}<br />
} else {<br />
next LOOP;<br />
}<br />
<br />
my (@ip) = split '\.', $addr;<br />
unless ($ip[0] == 69) { next LOOP; }<br />
unless ($ip[1] == 55) { next LOOP; }<br />
unless ($ip[2] == 229 || $ip[2] == 231) { next LOOP; }<br />
<br />
my $classC = "$ip[0]_$ip[1]_$ip[2]";<br />
<br />
# IP dir<br />
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }<br />
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }<br />
#<br />
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }<br />
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }<br />
#<br />
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }<br />
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }<br />
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];<br />
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];<br />
<br />
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];<br />
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];<br />
<br />
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];<br />
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];<br />
}<br />
}<br />
close(DATA);<br />
`rm /usr/home/working/tmp-$file`;<br />
&debug("processing ip totals at ");<br />
&debug(`date`);<br />
foreach my $classC (keys(%{$iptotal})) {<br />
my @values;<br />
foreach my $ip (keys(%{$iptotal->{$classC}})) {<br />
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {<br />
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};<br />
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};<br />
# $packets = $packets > 2147483647 ? 0 : $packets;<br />
if ($octets > 2147483647) {<br />
my $ddir = $dir==1 ? 'in' : 'out';<br />
#print SQL "$date $ip $ddir $octets\n";<br />
# $octets = 0;<br />
}<br />
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction<br />
my $id = "$ip-$condensedDate-$dir";<br />
$id =~ s/\.//g;<br />
push @values, "('$date', '$ip', $dir, $octets, $packets)";<br />
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";<br />
print "$sql\n" if $dry;<br />
print SQL "$sql;\n";<br />
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");<br />
}<br />
}<br />
<br />
# break inserts into 100 records at a time<br />
&debug("inserting $#values +1 values");<br />
while ($#values > 0) {<br />
my $sql = "insert into ipTotals_$classC values ";<br />
my $max_index = $#values > 100 ? 100 : $#values;<br />
for (my $i=0; $i<=$max_index; $i++) {<br />
$sql .= shift @values;<br />
$sql .= ',';<br />
}<br />
chop $sql;<br />
print "$sql\n" if $dry;<br />
print SQL "$sql;\n";<br />
}<br />
}<br />
<br />
# &debug("processing protocol totals at ");<br />
# &debug(`date`);<br />
# foreach my $classC (keys(%{$protototal})) {<br />
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;<br />
# my @values;<br />
# foreach my $ip (keys(%{$protototal->{$classC}})) {<br />
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {<br />
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {<br />
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};<br />
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};<br />
# # $octets = $octets > 2147483647 ? 0 : $octets;<br />
# # $packets = $packets > 2147483647 ? 0 : $packets;<br />
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto<br />
# my $id = "$ip-$condensedDate-$dir-$proto";<br />
# $id =~ s/\.//g;<br />
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";<br />
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";<br />
# print "$sql\n" if $dry;<br />
# $db->query($sql) unless $dry;<br />
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");<br />
# }<br />
# }<br />
# }<br />
# $db->query("unlock tables") unless $dry;<br />
# my $sql = "insert into protoTotals_$classC values ";<br />
# $sql .= join ',', @values;<br />
# $db->query("lock tables protoTotals_$classC write") unless $dry;<br />
# print "$sql\n" if $dry;<br />
# $db->query($sql) unless $dry;<br />
# $db->query("unlock tables") unless $dry;<br />
# }<br />
<br />
&debug("processing port totals at ");<br />
&debug(`date`);<br />
foreach my $classC (keys(%{$porttotal})) {<br />
my @values;<br />
foreach my $ip (keys(%{$porttotal->{$classC}})) {<br />
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {<br />
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {<br />
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {<br />
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};<br />
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};<br />
# $octets = $octets > 2147483647 ? 0 : $octets;<br />
# $packets = $packets > 2147483647 ? 0 : $packets;<br />
<br />
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port<br />
my $id = "$ip-$condensedDate-$dir-$proto-$port";<br />
$id =~ s/\.//g;<br />
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";<br />
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";<br />
print "$sql\n" if $dry;<br />
print SQL "$sql;\n";<br />
<br />
# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");<br />
}<br />
}<br />
}<br />
}<br />
<br />
# break inserts into 100 records at a time<br />
&debug("inserting $#values +1 values");<br />
while ($#values > 0) {<br />
my $sql = "insert into portTotals_$classC values ";<br />
my $max_index = $#values > 100 ? 100 : $#values;<br />
for (my $i=0; $i<=$max_index; $i++) {<br />
$sql .= shift @values;<br />
$sql .= ',';<br />
}<br />
chop $sql;<br />
print "$sql\n" if $dry;<br />
print SQL "$sql;\n";<br />
}<br />
}<br />
<br />
# 12 1 8 1 1= 23<br />
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction<br />
# 12 1 8 1 1 3=26<br />
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto<br />
# 12 1 8 1 1 5=28<br />
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port<br />
#print "finished at ";<br />
#print `date`;<br />
`mv $queuedir/$file $archivedir` unless $dry;<br />
close(SQL);<br />
`bzip2 $sqldirworking/$outfile`;<br />
`mv $sqldirworking/$outfile.bz2 $sqldir/`;<br />
}<br />
}<br />
`rm $queuedir/.lock` unless $dry;<br />
<br />
sub debug {<br />
my $message = shift;<br />
if ($debug) {<br />
print "$message\n";<br />
}<br />
}<br />
<br />
# var full during ft-v05.2005-03-28.084500-0800 and<br />
# 2005-02-24 69.55.226<br />
<br />
# all port/daily totals before 2005-04-07<br />
</pre><br />
<br />
This script sends the sql files to the traffic server for processing:<br />
<pre>cat > /usr/home/flowbin/sendsql.pl<br />
#!/usr/bin/perl<br />
<br />
#use strict;<br />
#$debug=1;<br />
#$dry=1;<br />
<br />
my $remote = "69.55.233.199";<br />
my $sqldir = "/usr/home/sql";<br />
my $archive = "/usr/home/archive";<br />
my $sqldirremote = "/data/bwdb2/pending/";<br />
my @err;<br />
unless ($dry) {<br />
if (-e "$sqldir/.lock") {<br />
open(FILE, "$sqldir/.lock");<br />
my $pid = <FILE>;<br />
chomp($pid);<br />
close(FILE);<br />
if (kill(0, $pid)) {<br />
#another process is using the queue, bail out<br />
exit(0);<br />
}<br />
else {<br />
#dead lock file, remove it<br />
`rm $sqldir/.lock`;<br />
}<br />
}<br />
open(FILE, "> $sqldir/.lock");<br />
print FILE "$$\n";<br />
close(FILE);<br />
}<br />
<br />
opendir(DIR, $sqldir);<br />
my @files = readdir(DIR);<br />
closedir(DIR);<br />
<br />
foreach my $file (sort @files) {<br />
next unless $file =~ /bz2$/;<br />
<br />
my $r = `scp -Cq $sqldir/$file $remote:$sqldirremote 2>&1`;<br />
# print "scp $sqldir/$file $remote:$sqldirremote";<br />
unless ($?==0) {<br />
push @err, "scp -Cq $sqldir/$file $remote:$sqldirremote ($r)";<br />
}<br />
else {<br />
`mv $sqldir/$file $archive`;<br />
`ssh $remote mv $sqldirremote/$file $sqldirremote/${file}.done`;<br />
}<br />
}<br />
<br />
`rm $sqldir/.lock` unless $dry;<br />
<br />
if (@err) {<br />
email_support('bwdb2: sendsql.pl error',join "\n", @err);<br />
}<br />
<br />
sub email_support {<br />
my $subj=shift;<br />
my $body=shift;<br />
use Mail::Sendmail;<br />
<br />
# prepare message<br />
my %mail = (<br />
To => 'support@johncompanies.com,dave@johncompanies.com',<br />
From => 'support@johncompanies.com',<br />
Subject => $subj,<br />
Message => $body,<br />
smtp => 'mail.johncompanies.com',<br />
);<br />
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";<br />
}<br />
<br />
sub debug {<br />
my $message = shift;<br />
if ($debug) {<br />
print "$message\n";<br />
}<br />
}<br />
<br />
# var full during ft-v05.2005-03-28.084500-0800 and<br />
# 2005-02-24 69.55.226<br />
<br />
# all port/daily totals before 2005-04-07<br />
</pre><br />
<br />
<pre>crontab -e<br />
#process flows<br />
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl<br />
#move sql commands to traffic db<br />
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl<br />
</pre><br />
<br />
==== flow processing: castle ====<br />
<pre><br />
cat > /usr/home/flowbin/processflows.pl<br />
<br />
#!/usr/bin/perl<br />
<br />
#use strict;<br />
#$debug=1;<br />
#$dry=1;<br />
<br />
my $log = '/usr/home/flowbin/discards.log';<br />
<br />
use Data::Dumper;<br />
<br />
BEGIN {<br />
push @INC, "/usr/home/flowbin";<br />
}<br />
<br />
use db;<br />
<br />
#my $queuedir = "/usr/home/queue";<br />
my $queuedir = "/usr/home/working";<br />
my $archivedir = "/usr/home/archive";<br />
<br />
unless ($dry) {<br />
if (-e "$queuedir/.lock") {<br />
open(FILE, "$queuedir/.lock");<br />
my $pid = <FILE>;<br />
chomp($pid);<br />
close(FILE);<br />
if (kill(0, $pid)) {<br />
#another process is using the queue, bail out<br />
exit(0);<br />
}<br />
else {<br />
#dead lock file, remove it<br />
`rm $queuedir/.lock`;<br />
}<br />
}<br />
open(FILE, "> $queuedir/.lock");<br />
print FILE "$$\n";<br />
close(FILE);<br />
}<br />
<br />
my $db = db->new();<br />
$db->connect('traffic', '', 'root', '5over3') || die $db->{'error'};<br />
<br />
opendir(DIR, $queuedir);<br />
my @files = readdir(DIR);<br />
closedir(DIR);<br />
<br />
foreach my $file (sort @files) {<br />
unless($file =~ /^\./) {<br />
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;<br />
my $date = "$1 $2:$3:$4";<br />
my $condensedDate = $1;<br />
$condensedDate =~ s/-//g;<br />
my $iptotal = {};<br />
my $protototal = {};<br />
my $porttotal = {};<br />
<br />
&debug("started file $file at ");<br />
&debug(`date`);<br />
&debug("getting raw flow data (flow-print)");<br />
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;<br />
&debug("aggregating data at ");<br />
&debug(`date`);<br />
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }<br />
LOOP: while (my $line = readline DATA) {<br />
my @d = split /[\s]+/, $line;<br />
if ($d[0] ne '' && $d[0] ne 'Start') {<br />
my $addr = 0;<br />
my $port = 0;<br />
<br />
#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets<br />
#0 1 2 3 4 5 6 7 8 9 10 11<br />
#|<br />
# outbound = 2, inbound = 1<br />
<br />
my (@src_ip) = split '\.', $d[3];<br />
my (@dst_ip) = split '\.', $d[6];<br />
<br />
if ($src_ip[0] == 69 && $src_ip[1] == 55 &&<br />
$src_ip[2] >= 224 && $src_ip[2] <= 239 &&<br />
$src_ip[2] != 229 && $src_ip[2] != 231) { # for castle<br />
# if ($src_ip[0] == 69 && $src_ip[1] == 55 && $src_ip[2] == 229) { # for i2b<br />
$d[2] = 2;<br />
# hack for outbound bulk traffic counted 2x<br />
if ($dst_ip[2] == 234) { $d[11] /= 2; $d[10] /= 2; }<br />
}<br />
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 &&<br />
$dst_ip[2] >= 224 && $dst_ip[2] <= 239 &&<br />
$dst_ip[2] != 229 && $dst_ip[2] != 231) { # for castle<br />
# elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && $dst_ip[2] == 229) { # for i2b<br />
$d[2] = 1;<br />
}<br />
else { next LOOP; }<br />
<br />
if ($d[2] == 2) {<br />
$addr = $d[3];<br />
# if the dst-port is low, store that<br />
if ($d[7] <= 1024) { $port = $d[7]; }<br />
# if the src-port is low, store that<br />
elsif ($d[4] <= 1024) { $port = $d[4]; }<br />
else { $port = 99999; }<br />
}<br />
elsif ($d[2] == 1) {<br />
$addr = $d[6];<br />
# if the dst-port is high, assume its return traffic, try to store src-port if low<br />
if ($d[7] > 1024) {<br />
if ($d[4] <= 1024) { $port = $d[4]; }<br />
else { $port = 99999; }<br />
} else {<br />
$port = $d[7];<br />
}<br />
} else {<br />
next LOOP;<br />
}<br />
<br />
my (@ip) = split '\.', $addr;<br />
unless ($ip[0] == 69) { next LOOP; }<br />
unless ($ip[1] == 55) { next LOOP; }<br />
unless ($ip[2] >= 224 && $ip[2] <= 239 && $ip[2] != 229 && $ip[2] != 231) { next LOOP; }<br />
# unless ($ip[2] == 229) { next LOOP; }<br />
<br />
my $classC = "$ip[0]_$ip[1]_$ip[2]";<br />
<br />
# IP dir<br />
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }<br />
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }<br />
#<br />
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }<br />
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }<br />
#<br />
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }<br />
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }<br />
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];<br />
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];<br />
<br />
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];<br />
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];<br />
<br />
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];<br />
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];<br />
}<br />
}<br />
close(DATA);<br />
`rm /usr/home/working/tmp-$file`;<br />
&debug("processing ip totals at ");<br />
&debug(`date`);<br />
foreach my $classC (keys(%{$iptotal})) {<br />
$db->query("lock tables dailyIpTotals_$classC write") unless $dry;<br />
my @values;<br />
foreach my $ip (keys(%{$iptotal->{$classC}})) {<br />
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {<br />
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};<br />
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};<br />
# $packets = $packets > 2147483647 ? 0 : $packets;<br />
if ($octets > 2147483647) {<br />
my $ddir = $dir==1 ? 'in' : 'out';<br />
`echo "$date $ip $ddir $octets\n" >> $log`;<br />
# $octets = 0;<br />
}<br />
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction<br />
my $id = "$ip-$condensedDate-$dir";<br />
$id =~ s/\.//g;<br />
push @values, "('$date', '$ip', $dir, $octets, $packets)";<br />
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";<br />
print "$sql\n" if $dry;<br />
$db->query($sql) unless $dry;<br />
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");<br />
}<br />
}<br />
$db->query("unlock tables") unless $dry;<br />
<br />
$db->query("lock tables ipTotals_$classC write") unless $dry;<br />
# break inserts into 100 records at a time<br />
&debug("inserting $#values +1 values");<br />
while ($#values > 0) {<br />
my $sql = "insert into ipTotals_$classC values ";<br />
my $max_index = $#values > 100 ? 100 : $#values;<br />
for (my $i=0; $i<=$max_index; $i++) {<br />
$sql .= shift @values;<br />
$sql .= ',';<br />
}<br />
chop $sql;<br />
print "$sql\n" if $dry;<br />
$db->query($sql) unless $dry;<br />
}<br />
$db->query("unlock tables") unless $dry;<br />
}<br />
<br />
sleep 20;<br />
# &debug("processing protocol totals at ");<br />
# &debug(`date`);<br />
# foreach my $classC (keys(%{$protototal})) {<br />
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;<br />
# my @values;<br />
# foreach my $ip (keys(%{$protototal->{$classC}})) {<br />
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {<br />
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {<br />
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};<br />
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};<br />
# # $octets = $octets > 2147483647 ? 0 : $octets;<br />
# # $packets = $packets > 2147483647 ? 0 : $packets;<br />
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto<br />
# my $id = "$ip-$condensedDate-$dir-$proto";<br />
# $id =~ s/\.//g;<br />
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";<br />
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";<br />
# print "$sql\n" if $dry;<br />
# $db->query($sql) unless $dry;<br />
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");<br />
# }<br />
# }<br />
# }<br />
# $db->query("unlock tables") unless $dry;<br />
# my $sql = "insert into protoTotals_$classC values ";<br />
# $sql .= join ',', @values;<br />
# $db->query("lock tables protoTotals_$classC write") unless $dry;<br />
# print "$sql\n" if $dry;<br />
# $db->query($sql) unless $dry;<br />
# $db->query("unlock tables") unless $dry;<br />
# }<br />
<br />
&debug("processing port totals at ");<br />
&debug(`date`);<br />
foreach my $classC (keys(%{$porttotal})) {<br />
$db->query("lock tables dailyPortTotals_$classC write") unless $dry;<br />
my @values;<br />
foreach my $ip (keys(%{$porttotal->{$classC}})) {<br />
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {<br />
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {<br />
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {<br />
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};<br />
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};<br />
# $octets = $octets > 2147483647 ? 0 : $octets;<br />
# $packets = $packets > 2147483647 ? 0 : $packets;<br />
<br />
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port<br />
my $id = "$ip-$condensedDate-$dir-$proto-$port";<br />
$id =~ s/\.//g;<br />
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";<br />
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";<br />
print "$sql\n" if $dry;<br />
$db->query($sql) unless $dry;<br />
# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");<br />
}<br />
}<br />
}<br />
}<br />
$db->query("unlock tables") unless $dry;<br />
<br />
$db->query("lock tables portTotals_$classC write") unless $dry;<br />
# break inserts into 100 records at a time<br />
&debug("inserting $#values +1 values");<br />
while ($#values > 0) {<br />
my $sql = "insert into portTotals_$classC values ";<br />
my $max_index = $#values > 100 ? 100 : $#values;<br />
for (my $i=0; $i<=$max_index; $i++) {<br />
$sql .= shift @values;<br />
$sql .= ',';<br />
}<br />
chop $sql;<br />
print "$sql\n" if $dry;<br />
$db->query($sql) unless $dry;<br />
}<br />
$db->query("unlock tables") unless $dry;<br />
sleep 10;<br />
}<br />
<br />
# 12 1 8 1 1= 23<br />
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction<br />
# 12 1 8 1 1 3=26<br />
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto<br />
# 12 1 8 1 1 5=28<br />
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port<br />
#print "finished at ";<br />
#print `date`;<br />
`mv $queuedir/$file $archivedir` unless $dry;<br />
}<br />
}<br />
`rm $queuedir/.lock` unless $dry;<br />
<br />
sub debug {<br />
my $message = shift;<br />
if ($debug) {<br />
print "$message\n";<br />
}<br />
}<br />
<br />
# var full during ft-v05.2005-03-28.084500-0800 and<br />
# 2005-02-24 69.55.226<br />
<br />
# all port/daily totals before 2005-04-07<br />
</pre><br />
<br />
<pre><br />
cat > /usr/home/flowbin/db.pm<br />
<br />
#!/usr/bin/perl<br />
#<br />
# $Header: /usr/cvs/newgw/lib/db.pm,v 1.4 2003/06/05 18:20:01 glenn Exp $<br />
#<br />
# Copyright (c) 2003<br />
# e-Monitoring Networks, Inc. All rights reserved.<br />
#<br />
#<br />
package db;<br />
<br />
use strict;<br />
use DBI;<br />
<br />
sub new {<br />
my $class = shift;<br />
my $self = {};<br />
<br />
$self->{'debug'} = 0;<br />
bless $self, $class;<br />
<br />
return $self;<br />
}<br />
<br />
sub connect {<br />
my $self = shift;<br />
my $dbname = shift;<br />
my $dbhost = shift;<br />
my $dbuser = shift;<br />
my $dbpass = shift;<br />
<br />
my $host = '';<br />
if (defined($dbhost)) {<br />
$host = ";host=$dbhost";<br />
}<br />
<br />
eval {<br />
$self->debug("connecting to: DBI:mysql:database=$dbname;$host", 1);<br />
$self->{'dbh'} = DBI->connect("DBI:mysql:database=$dbname;$host", $dbuser, $dbpass);<br />
};<br />
if ($self->{'dbh'}) {<br />
return 1;<br />
}<br />
$self->{'error'} = "Error connecting to database $@";<br />
$self->debug("Error connecting to database $@");<br />
return 0;<br />
}<br />
<br />
sub query {<br />
my $self = shift;<br />
my $query = shift;<br />
<br />
$self->debug($query, 1);<br />
my $sth;<br />
eval {<br />
$sth = $self->{'dbh'}->prepare($query);<br />
};<br />
unless ($sth) {<br />
$self->{'error'} = "error preparing query $@";<br />
$self->debug("error preparing query $@");<br />
return undef;<br />
}<br />
my $qty;<br />
eval {<br />
$qty = $sth->execute;<br />
};<br />
unless ($qty) {<br />
$self->{'error'} = "error executing query $@";<br />
warn "error executing query $@ $query";<br />
return undef;<br />
}<br />
$self->debug("returning $qty, $sth from query", 6);<br />
return ($qty, $sth);<br />
}<br />
<br />
sub disconnect {<br />
my $self = shift;<br />
<br />
$self->{'dbh'}->disconnect;<br />
return 0;<br />
}<br />
<br />
sub debug {<br />
my $self = shift;<br />
my $msg = shift;<br />
my $level = shift || 0;<br />
<br />
if ($level < $self->{'debug'}) {<br />
print "$msg\n";<br />
}<br />
return 0;<br />
}<br />
1;<br />
</pre><br />
<br />
mkdir /usr/home/archive<br />
mkdir -p /usr/home/sql/tmp<br />
<br />
<pre>crontab -e<br />
#process flows<br />
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl</pre><br />
<br />
==== setup traffic db ====<br />
* Install mysql:<br />
<pre>cd /usr/ports/databases/mysql50-server<br />
make install clean</pre><br />
<br />
cat >> /etc/rc.conf<br />
mysql_enable="YES" <br />
<br />
Move db data dir:<br />
/usr/local/etc/rc.d/mysql-server stop<br />
mkdir /usr/home/database/<br />
mv /var/db/mysql/* /usr/home/database/<br />
chown -R mysql:mysql /usr/home/database<br />
<br />
Edit database location in startup script:<br />
vi /usr/local/etc/rc.d/mysql-server<br />
# : ${mysql_dbdir="/var/db/mysql"}<br />
: ${mysql_dbdir="/usr/home/database"}<br />
<br />
/usr/local/etc/rc.d/mysql-server start<br />
<br />
<br />
* Install mysql perl database modules:<br />
<pre><br />
cd /usr/ports/databases/p5-DBI<br />
make install clean<br />
cd /usr/ports/databases/p5-DBD-mysql50<br />
make install clean<br />
(no to SSL support)<br />
</pre><br />
<br />
* Setting up database<br />
<pre><br />
rehash<br />
/usr/local/etc/rc.d/mysql-server start<br />
mysql -u root<br />
create database traffic;<br />
grant all on *.* to root@localhost identified by '5over3';<br />
grant all on traffic.* to jc@10.1.4.5 identified by '2gMKY3Wt';<br />
<br />
</pre><br />
<br />
If this was a new server we'd setup new tables. See [[#mysql_2|mysql]] for how those tables would be setup.<br />
<br />
We are assuming here we are moving data from an existing db, here's how that's done (from the current traffic db):<br />
rsync -av --progress /usr/home/database/traffic/ 10.1.4.203:/usr/home/database/traffic/<br />
<br />
When you're ready to do the cutover, shut down mysql on both hosts and do one last sync.<br />
<br />
<br />
<br />
==== process flows from bwdb2 ====<br />
On traffic database server (bwdb):<br />
<br />
<pre>crontab -e<br />
#import sql from bwdb2<br />
10,25,40,55 * * * * /usr/home/flowbin/processsql.pl</pre><br />
<br />
Add access to mysql: <br />
<pre>mysql -u root -p <br />
grant all on traffic.* to bwdb2@localhost identified by 's1lver4d';<br />
</pre> <br />
<br />
<pre>cat > /usr/home/flowbin/processsql.pl<br />
<br />
#!/usr/bin/perl<br />
<br />
#use strict;<br />
#$debug=1;<br />
#$dry=1;<br />
<br />
my $sqldir = "/usr/home/bwdb2/pending";<br />
my $mysql = '/usr/local/bin/mysql';<br />
my @err;<br />
unless ($dry) {<br />
if (-e "$sqldir/.lock") {<br />
open(FILE, "$sqldir/.lock");<br />
my $pid = <FILE>;<br />
chomp($pid);<br />
close(FILE);<br />
if (kill(0, $pid)) {<br />
#another process is using the queue, bail out<br />
exit(0);<br />
}<br />
else {<br />
#dead lock file, remove it<br />
`rm $sqldir/.lock`;<br />
}<br />
}<br />
open(FILE, "> $sqldir/.lock");<br />
print FILE "$$\n";<br />
close(FILE);<br />
}<br />
<br />
opendir(DIR, $sqldir);<br />
my @files = readdir(DIR);<br />
closedir(DIR);<br />
<br />
foreach my $file (sort @files) {<br />
next unless $file =~ /done$/;<br />
my $r = `bzcat $sqldir/$file | $mysql -u bwdb2 -ps1lver4d traffic`;<br />
unless ($?==0) {<br />
push @err, "bzcat $sqldir/$file | $mysql -u bwdb2 -pxxxxx traffic ($r)";<br />
}<br />
else {<br />
`rm $sqldir/$file`;<br />
}<br />
}<br />
<br />
`rm $sqldir/.lock` unless $dry;<br />
<br />
if (@err) {<br />
email_support('bwdb: processsql.pl error',join "\n", @err);<br />
}<br />
<br />
sub email_support {<br />
my $subj=shift;<br />
my $body=shift;<br />
use Mail::Sendmail;<br />
<br />
# prepare message<br />
my %mail = (<br />
To => 'dave@johncompanies.com',<br />
From => 'support@johncompanies.com',<br />
Subject => $subj,<br />
Message => $body,<br />
smtp => 'mail.johncompanies.com',<br />
);<br />
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";<br />
}<br />
<br />
sub debug {<br />
my $message = shift;<br />
if ($debug) {<br />
print "$message\n";<br />
}<br />
}<br />
</pre><br />
<br />
chmod 0700 /usr/home/flowbin/processsql.pl<br />
<br />
Make sure bwdb is reachable from the outside only to bwdb2:<br />
<br />
On nat, add to <tt>/etc/ipnat.rules</tt><br />
<pre># bwdb<br />
bimap fxp0 10.1.4.203/32 -> 69.55.233.199/32</pre><br />
<br />
Reload:<br />
ipnat -C -F -f /etc/ipnat.rules<br />
<br />
Setup firewall rule on firewall:<br />
ipfw add 00094 allow ip from 66.181.18.5 to 69.55.233.199 22<br />
ipfw add 00094 deny ip from any to 69.55.233.199<br />
<br />
Setup firewall on bwdb to restrict access now that it's nat'd:<br />
<pre><br />
cat >> /usr/local/etc/rc.d/boot.sh<br />
ipfw add 1 allow tcp from any to any established<br />
ipfw add 2 allow ip from 10.1.4.0/24,66.181.18.5,69.55.233.195 to me 22<br />
ipfw add 3 allow ip from 10.1.4.5 to me 3306<br />
ipfw add 4 allow ip from 69.55.225.225 53 to me <br />
ipfw add 5 allow ip from 69.55.230.2 25 to me <br />
ipfw add 6 allow ip from me to me 4444<br />
ipfw add 7 allow icmp from any to me<br />
ipfw add 8 allow udp from 10.1.4.203 to 10.1.4.203 dst-port 4444<br />
ipfw add 9 allow udp from 10.1.4.5 to me 161<br />
ipfw add 100 deny ip from any to me<br />
</pre> <br />
<br />
chmod 0700 /usr/local/etc/rc.d/boot.sh<br />
<br />
From bwdb2, add ssh key:<br />
cat /root/.ssh/id_dsa.pub | ssh 69.55.233.199 'cat - >> /root/.ssh/authorized_keys' <br />
<br />
Confirm no password access:<br />
ssh 69.55.233.199 hostname<br />
<br />
= bwdb2 =<br />
== Summary ==<br />
This machine tracks and stores network traffic (netflow) at i2b. It is our means to monitor customer bandwidth usage.<br />
<br />
* Location: i2b, cab6<br />
* OS: FreeBSD 6.4 x86<br />
* Networking: Priv IP: 10.1.2.4 There are 2 onboard nic's, one of which is the "listener"<br />
* Hardware: Custom 2U. Single power supply. <br />
* Drives: two 150 GB (2 x 150GB) RAID1 arrays running on a 3ware 7006 RAID card.<br />
<br />
== Services Provided ==<br />
* netflow<br />
* bigbrother <br />
<br />
== netflow ==<br />
<br />
The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.<br />
<br />
A cronjob moves that flow file (or files if there are multiple due to some delay) <br />
1,16,31,46 * * * * /usr/home/flowbin/queue.pl<br />
<br />
into a processing queue:<br />
<tt>/usr/home/working</tt><br />
<br />
Then a separate file processes whatever flow files it finds there, and builds sql files ready for insertion into the traffic database:<br />
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl<br />
<br />
Then yet another process copies the sql files to the traffic database server for processing and insertion into the mysql database:<br />
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl<br />
<br />
== Regular maintenance ==<br />
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]<br />
<br />
* if space becomes tight, move sql files and flow files to backup server, both located in <tt>/usr/home/flowbin/archive</tt><br />
<br />
= firewall (newgateway) =<br />
<br />
== Summary ==<br />
<br />
This machine is the primary (only) firewall for the entire network at castle. <br />
<br />
* Location: castle, cab 3-8<br />
* OS: FreeBSD 4.11 x86<br />
* Networking: Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).<br />
* Hardware: 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card.<br />
<br />
== Services Provided ==<br />
* firewall (ipfw)<br />
* snmp<br />
* bigbrother<br />
<br />
== Firewall Rule Configuration ==<br />
<br />
See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.<br />
<br />
== Disaster Recovery ==<br />
<br />
If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.<br />
<br />
Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)<br />
<br />
<pre>hostname="newgateway.johncompanies.com"<br />
firewall_script="/etc/firewall.sh"<br />
firewall_enable="NO"<br />
sendmail_enable="NONE"<br />
sshd_enable="YES"<br />
inetd_enable="NO"<br />
xntpd_enable="YES"<br />
snmpd_enable="YES"<br />
#snmpd_flags="-as -p /var/run/snmpd.pid"<br />
#ipnat_enable="YES"<br />
#ipnat_rules="/etc/ipnat.rules"<br />
gateway_enable="YES"<br />
<br />
defaultrouter="69.55.233.161"<br />
<br />
ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0"<br />
ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248"<br />
<br />
#<br />
# Original JohnCompanies 69.55.224.0/20<br />
#<br />
ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248"<br />
<br />
static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1<br />
2 route13 route14 route15 route16 route17 route18"<br />
<br />
route_route1="-net 69.55.224.0 69.55.233.153"<br />
route_route2="-net 69.55.225.0 69.55.233.153"<br />
route_route3="-net 69.55.226.0 69.55.233.153"<br />
route_route4="-net 69.55.227.0 69.55.233.153"<br />
route_route5="-net 69.55.228.0 69.55.233.153"<br />
route_route6="-net 69.55.229.0 69.55.233.153"<br />
route_route7="-net 69.55.230.0 69.55.233.153"<br />
route_route8="-net 69.55.231.0 69.55.233.153"<br />
route_route9="-net 69.55.232.0 69.55.233.153"<br />
route_route10="-net 69.55.233.0 69.55.233.153"<br />
route_route11="-net 69.55.234.0 69.55.233.153"<br />
route_route12="-net 69.55.235.0 69.55.233.153"<br />
route_route13="-net 69.55.236.0 69.55.233.153"<br />
route_route14="-net 69.55.237.0 69.55.233.153"<br />
route_route15="-net 69.55.238.0 69.55.233.153"<br />
route_route16="-net 69.55.239.0 69.55.233.153"<br />
route_route17="-net 10.1.5.0 10.1.4.2"<br />
route_route18="-net 10.1.6.0 10.1.4.2"<br />
<br />
<br />
#In case of 3750 failure:<br />
#defaultrouter="69.43.128.81"<br />
#ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248"<br />
<br />
#bind .1's here:<br />
#ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0"<br />
#ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0"<br />
<br />
#bulk:<br />
# reassign 69.55.231.1 to the int iface on the firewall<br />
# set the DG on the firewall to 69.43.138.9<br />
# set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248</pre><br />
<br />
== Cronjobs ==<br />
1 0 * * * /usr/local/etc/rsync.backup<br />
Backup to backup1<br />
<br />
0 0 1 * * /sbin/ipfw zero<br />
0 0 1 * * /sbin/ipfw del 3 4 5 17331<br />
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap). <br />
<br />
Inside <tt>/etc/daily.local</tt> you will see a call to <tt>/etc/makepiperules.pl</tt><br />
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.<br />
<br />
== DOS attacks ==<br />
<br />
See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.<br />
<br />
Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will<br />
# enter a rule that allows all UDP to go through<br />
# send an emergency email to support and indicating an attack is in progress<br />
# send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted<br />
# wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1<br />
This file lives under /usr/home/user/doswatch.pl<br />
To run:<br />
cd /usr/home/user<br />
./doswatch.pl &<br />
<br />
To kill;<br />
fg<br />
^C<br />
<br />
It writes its findings to /usr/home/user/doswatch.log<br />
<br />
= backup1 =<br />
<br />
== Summary ==<br />
<br />
This machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server. <br />
<br />
* Location: castle, cab 3-8<br />
* OS: Ubuntu 8.04.1 server x86<br />
* Networking: Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)<br />
* Hardware: 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Single power supply. <br />
* Drives: 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card<br />
<br />
== Services provided ==<br />
* backup via rsync<br />
* mysql - traffic data<br />
* nfs server - for backups<br />
* snmp client - for big brother<br />
* bigbrother client<br />
<br />
== Usage and Notes ==<br />
* all data is stored under /data<br />
* virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: <tt>/backup1</tt>), and they all have their ssh keys setup to allow passwordless rsync's<br />
* each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)<br />
* at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.<br />
* cancelled customer systems are compressed and stored under <tt>/data/deprecated</tt><br />
* archived bwdb2 flow files are stored under <tt>/data/bwdb2</tt><br />
* critical files from backup2 are stored under <tt>/data/backup2</tt><br />
<br />
== Cronjobs ==<br />
<pre><br />
00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null<br />
35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate /data/backuplog.log</pre><br />
this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup. It will email support@johncompanies.com at it’s conclusion. This email can be deleted, however note when it begins to take significantly longer to complete, ie runs past 2200 pm – this usually indicates a problem on the backup server.<br />
<br />
<pre>10,25,40,55 * * * * /usr/local/sbin/processsql.pl<br />
</pre><br />
this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.<br />
<pre>0 0 * * * /usr/local/sbin/3wraidchk<br />
</pre><br />
checks the health of the RAID array<br />
<br />
== Regular maintenance ==<br />
*[[Routine_Maintenance#Free_up_space_on_backup1|Remove old backups]]<br />
*[[Routine_Maintenance#3ware|Check on auto-verify]]<br />
<br />
== build ==<br />
<br />
<pre>Setup raid5 with a boot vol of 12G 5.45tb <br />
12G boot<br />
4664 GB <br />
<br />
Install ubuntu 8.04<br />
<br />
Swap 4G<br />
<br />
Don’t format data drive<br />
<br />
http://www.unixgods.org/~tilo/linux_larger_2TB.html<br />
<br />
parted /dev/sdb<br />
print<br />
mklabel gpt<br />
print<br />
<br />
#Disk /dev/sdb: 4987GB<br />
#Sector size (logical/physical): 512B/512B<br />
#Partition Table: gpt<br />
<br />
#Number Start End Size File system Name Flags<br />
<br />
mkpart primary ext3 0 4987GB<br />
print<br />
<br />
#Disk /dev/sdb: 5987GB<br />
#Sector size (logical/physical): 512B/512B<br />
#Partition Table: gpt<br />
<br />
#Number Start End Size File system Name Flags<br />
# 1 17.4kB 4987GB 4987GB primary<br />
<br />
quit<br />
<br />
mkfs.ext3 /dev/sdb1<br />
#mke2fs 1.40.8 (13-Mar-2008)<br />
#Filesystem label=<br />
#OS type: Linux<br />
#Block size=4096 (log=2)<br />
#Fragment size=4096 (log=2)<br />
#304390144 inodes, 1217544183 blocks<br />
#60877209 blocks (5.00%) reserved for the super user<br />
#First data block=0<br />
#Maximum filesystem blocks=0<br />
#37157 block groups<br />
#32768 blocks per group, 32768 fragments per group<br />
#8192 inodes per group<br />
#Superblock backups stored on blocks:<br />
# 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,<br />
# 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,<br />
# 102400000, 214990848, 512000000, 550731776, 644972544<br />
#<br />
#Writing inode tables: 967/37157<br />
<br />
mkdir /data<br />
<br />
#root@backup1:~# df -h<br />
#Filesystem Size Used Avail Use% Mounted on<br />
#/dev/sda2 8.3G 540M 7.3G 7% /<br />
#varrun 1013M 40K 1013M 1% /var/run<br />
#varlock 1013M 0 1013M 0% /var/lock<br />
#udev 1013M 56K 1013M 1% /dev<br />
#devshm 1013M 0 1013M 0% /dev/shm<br />
#/dev/sdb1 4.5T 192M 4.3T 1% /data<br />
<br />
<br />
apt-get update<br />
apt-get upgrade<br />
apt-get install snmp snmpd ntp nfs-kernel-server<br />
<br />
echo "\"\e[5~\": history-search-backward" >> ~/.inputrc<br />
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc<br />
<br />
vi /etc/ntp.conf<br />
server 10.1.4.5<br />
<br />
scp root@10.1.4.3:/root/.ssh/authorized_keys /root/.ssh/<br />
cd /root/<br />
ssh-keygen -t dsa<br />
echo "10.1.4.3 backup2" >> /etc/hosts<br />
<br />
cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' <br />
<br />
ssh backup2<br />
<br />
vi /root/.bashrc<br />
export PS1="[\u@\h \w]# "<br />
alias h='history'<br />
alias vi='vim'<br />
alias j='jobs'<br />
export PS1="[\u@\h \w]# "<br />
alias dr='screen -dr'<br />
export EDITOR=vim<br />
export GREP_OPTIONS='--color=auto'<br />
export HISTFILESIZE=1000<br />
<br />
source /root/.bashrc<br />
<br />
echo "# ttyS0 - getty <br />
# <br />
# This service maintains a getty on ttyS0 from the point the system is<br />
# started until it is shut down again.<br />
<br />
start on runlevel 2<br />
start on runlevel 3<br />
start on runlevel 4<br />
start on runlevel 5<br />
<br />
stop on runlevel 0<br />
stop on runlevel 1<br />
stop on runlevel 6<br />
<br />
respawn<br />
exec /sbin/getty 38400 ttyS0" > /etc/event.d/ttyS0 <br />
<br />
<br />
vi /boot/grub/menu.lst<br />
<br />
serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1<br />
terminal --timeout=15 serial console<br />
<br />
append to kernel lines: <br />
console=tty0 console=ttyS0,38400n8<br />
<br />
show menu:<br />
#hiddenmenu<br />
<br />
echo 'rocommunity jcread 10.1.4.5<br />
rocommunity jcread 10.1.4.3<br />
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf<br />
<br />
# to see which iface it is, on backup2:<br />
<br />
snmpwalk -v 1 -c jcread 10.1.4.8 interface<br />
<br />
<br />
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd<br />
<br />
echo "bb:x:1984:" >> /etc/group<br />
<br />
pwconv<br />
<br />
mkdir /home/bb<br />
chown bb.bb /home/bb<br />
<br />
cd ~bb<br />
scp backup2:/mnt/data4/build/bb/bb-linux.tar .<br />
<br />
tar xf bb-linux.tar<br />
<br />
cd /home/bb/bbc1.9e-btf/etc<br />
<br />
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts<br />
echo "10.1.4.8 backup1.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts<br />
<br />
echo "/:90:95<br />
/var:90:95<br />
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab<br />
<br />
<br />
vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh<br />
(remove all | SORT xxxx)<br />
<br />
chmod +r /var/log/messages<br />
<br />
./bbchkcfg.sh <br />
#(y to questions)<br />
./bbchkhosts.sh<br />
#(ignore ssh errors)<br />
cd ../..<br />
chown -R bb .<br />
su bb<br />
cd<br />
cd bbc1.9e-btf/src<br />
<br />
#make; make install<br />
cd ..<br />
./runbb.sh start<br />
more BBOUT <br />
(look for errors)<br />
exit<br />
<br />
vi /etc/rc.local<br />
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"<br />
<br />
<br />
echo '/data 10.1.4.0/24(rw, no_root_squash,async,no_subtree_check)' >> /etc/exports<br />
<br />
/etc/init.d/nfs-kernel-server restart<br />
<br />
<br />
echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd<br />
<br />
<br />
echo '10.1.4.8 backup1' >> /etc/hosts<br />
echo '/dev/sdb1 /data ext3 rw,noatime 0 0' >> /etc/fstab<br />
<br />
<br />
to install digi drivers:<br />
<br />
wget http://ftp1.digi.com/support/driver/40002086_n.tgz<br />
apt-get install linux-image-2.6.24-19-server<br />
apt-get install linux-source-2.6.24 (not needed?)<br />
apt-get install linux-headers-2.6.24-19-server <br />
apt-get install make<br />
apt-get install gcc<br />
apt-get install g++ <br />
apt-get install libncurses5-dev<br />
apt-get install expect<br />
apt-get install libdbi-perl libdate-calc-perl libdbd-mysql-perl<br />
<br />
cd /usr/src; ln -s linux-headers-2.6.24-19-server linux<br />
./configure<br />
make all<br />
make install<br />
make postinstall<br />
<br />
/usr/bin/dgrp_cfg_node -v -v init el 65.116.11.2 8<br />
<br />
apt-get install mysql<br />
<br />
mkdir /data/mysql<br />
chown mysql:mysql /data/mysql<br />
/etc/init.d/mysql stop<br />
mv /var/lib/mysql/* /data/mysql/<br />
mv /data/mysql/ib_* /var/lib/mysql/<br />
vi /etc/mysql/my.cnf<br />
(change datadir to /data/mysql)<br />
vi /etc/apparmor.d/usr.sbin.mysqld<br />
add:<br />
/data/mysql/ r,<br />
/data/mysql/** rwk,<br />
Comment out:<br />
# /var/lib/mysql/ r,<br />
# /var/lib/mysql/** rwk,<br />
<br />
/etc/init.d/apparmor restart<br />
/etc/init.d/mysql start<br />
<br />
tw_cli /c0/u0 set ignoreECC=on<br />
tw_cli /c0/u0 set storsave=balance<br />
tw_cli /c0/u0 set cache=on<br />
<br />
<br />
0 0 * * * /usr/local/sbin/3wraidchk<br />
</pre><br />
<br />
= backup2 =<br />
<br />
== Summary ==<br />
<br />
This machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. NOTE: power button is broken, so the reset button (paper clip) was rewired to be the power button.<br />
<br />
* Location: castle, cab 3-7<br />
* OS: FreeBSD 6.1 x86<br />
* Networking: Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC infrastructure @ i2b)<br />
* Hardware: 16 IDE drive bays (4 columns of 4, drive 0-0 top left, drive 0-1 just to the right TODO) all hot-swap. Triple power supply. <br />
* Drives: <br />
**3ware 7500-8:<br />
***200 GB JBOD (1 x 200G) labeled 0-0<br />
***500 GB RAID5 (3 x 250G) 0-1 thru 0-3<br />
***700 GB RAID5 (4 x 250G) 0-4 thru 0-7<br />
**3ware 7500-8:<br />
***700 GB RAID5 (4 x 250G) 1-0 thru 1-3<br />
***700 GB RAID5 (4 x 250G) 1-4 thru 1-7<br />
<br />
All drives MUST be western digital IDE drives. Other brands will not fit.<br />
<br />
In case of an outage, nfs will hang on all connected servers until the nfs service returns. If you can't get backup2 back online, you can get nfs running elsewhere and fake backup2's MAC's: priv: 00:0e:0c:59:c1:a6, pub: 00:07:e9:5b:c6:45<br />
<br />
To configure:<br />
ifconfig fxp0 link 00:90:27:f9:0a:d9<br />
<br />
== Services provided ==<br />
* backup via rsync and nfs<br />
* samba<br />
* nfs<br />
* snmp<br />
* bigbrother<br />
<br />
== Usage ==<br />
* all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: <tt>/mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4</tt><br />
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/mnt/data2/iso</tt><br />
* this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data<br />
* this server serves as an archive for exported db data from bwdb and old flow files.<br />
* isys backs up here<br />
* customers are nfs-moutned under /mnt/data3/customers as file-backed md devices<br />
* in <tt>/mnt/data4</tt> there are lots of useful things used for building our vps servers, customer servers, and management scripts:<br />
** <tt>/bin</tt>: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)<br />
** <tt>/build</tt>: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on<br />
** <tt>/vzrpms</tt>: contains the OS templates for many-to-most of the OS's we offer on vz systems<br />
<br />
== Cronjobs ==<br />
* backs itself up nightly to nfs-mounted backup1 (mountpoint: <tt>/backup2</tt>)<br />
<br />
== Regular maintenance ==<br />
*[[Routine_Maintenance#3ware|Check on health]]<br />
<br />
= backup3 =<br />
== Summary ==<br />
This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups. <br />
<br />
* Location: i2b, cab 6<br />
* OS: Ubuntu 10.04.1 server amd64<br />
* Networking: Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2<br />
* Hardware: 16 drive SATA bays (4 columns of 4, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: 5 TB (6 x 1TB) RAID5 array running on an Areca Technology Corp. ARC-1160 16-Port<br />
<br />
== Services provided ==<br />
* backup via rsync and nfs<br />
* samba<br />
* nfs<br />
* digi realport<br />
* snmp<br />
* bigbrother<br />
<br />
== Usage ==<br />
* all data is stored under /data<br />
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt><br />
* this server serves as an archive for exported db data from bwdb and old flow files.<br />
* inftrastructure machines at i2b back up here<br />
* customers are nfs-moutned under /data/customers as file-backed loopback devices<br />
<br />
== management scripts ==<br />
* mkbackups<br />
<br />
mkbackup <cid> GB <ip><br />
<br />
== Cronjobs ==<br />
0 0 * * * /usr/local/sbin/arecaraidchk<br />
RAID checks<br />
<br />
35 4 * * * /usr/local/sbin/snapshot_archive<br />
Rotate daily snapshots for infrastructure machine backups<br />
<br />
== Regular maintenance ==<br />
*[[Routine_Maintenance#Areca|Check on RAID health]]<br />
<br />
== Build ==<br />
<br />
=== BIOS Config ===<br />
disable quiet boot<br />
<br />
set to last state after power loss<br />
<br />
set date/time to GMT<br />
<br />
enable serial console output (baud rate 115200)<br />
<br />
=== Install OS ===<br />
<pre>Ubuntu 10.04.1 amd64 (couldn't get 12.04 to load cause the H/W was incompat)<br />
10G / ext3<br />
2G swap<br />
~ /data ext4<br />
<br />
Install packages:<br />
openssh<br />
samba</pre><br />
<br />
=== DNS and private IP ===<br />
<br />
echo "nameserver 69.55.225.225" >> /etc/resolv.conf<br />
<br />
Add a 2nd IP to eth0 and setup priv net<br />
<pre>vi /etc/network/interfaces<br />
<br />
auto eth0<br />
iface eth0 inet static<br />
address 69.55.229.4<br />
netmask 255.255.255.0<br />
network 69.55.229.0<br />
broadcast 69.55.229.255<br />
gateway 69.55.229.1<br />
# dns-* options are implemented by the resolvconf package, if installed<br />
dns-nameservers 69.55.229.3 66.181.0.2<br />
dns-search johncompanies.com<br />
<br />
auto eth0:1<br />
iface eth0:1 inet static<br />
address 69.55.231.2<br />
netmask 255.255.255.0<br />
network 69.55.231.0<br />
broadcast 69.55.231.255<br />
<br />
auto eth1<br />
iface eth1 inet static<br />
address 10.1.2.3<br />
netmask 255.255.255.0<br />
network 10.1.2.0<br />
broadcast 10.1.2.255<br />
<br />
</pre><br />
<br />
=== Install packages ===<br />
<pre>apt-get update<br />
apt-get upgrade<br />
apt-get install gcc<br />
apt-get install libssl-dev<br />
apt-get install libncurses5-dev<br />
apt-get install cu<br />
apt-get install unzip<br />
apt-get install snmp snmpd ntp nfs-kernel-server</pre><br />
<br />
=== tweak grub, enable serial ===<br />
<br />
<pre>vi /etc/default/grub<br />
#GRUB_HIDDEN_TIMEOUT=0<br />
GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64"<br />
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"<br />
update-grub</pre><br />
<br />
<pre>echo "start on stopped rc RUNLEVEL=[2345]<br />
stop on runlevel [!2345]<br />
respawn<br />
exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf</pre><br />
<br />
=== install realport (digi) driver ===<br />
<br />
give the digi an ip with DgIpServ.exe<br />
<br />
<pre>cd /usr/src/<br />
wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz<br />
tar xzf dgrp-1.9.tgz <br />
cd dgrp-1.9/<br />
./configure<br />
make<br />
make install<br />
make postinstall<br />
update-rc.d dgrp_daemon defaults</pre><br />
<br />
configure ports:<br />
dgrp_cfg_node init el 10.1.2.10 16<br />
<br />
try connecting with:<br />
cu -l /dev/ttyel00 -s 38400<br />
<br />
=== shell, ntp, ssh key, hosts ===<br />
<br />
Shell autocompletion search:<br />
<pre>echo "\"\e[5~\": history-search-backward" >> ~/.inputrc<br />
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre><br />
<br />
Setup ntp:<br />
<pre>vi /etc/ntp.conf<br />
server 10.1.2.1<br />
server ntp.ubuntu.com</pre><br />
<br />
Generate ssh keys:<br />
<pre>cd /root/<br />
ssh-keygen -t dsa</pre><br />
Defaults, no password<br />
<br />
Setup hosts:<br />
<pre>echo "69.55.230.10 backup2" >> /etc/hosts<br />
echo "69.55.230.11 backup1" >> /etc/hosts<br />
echo "10.1.2.4 bwdb2" >> /etc/hosts<br />
echo "10.1.2.3 backup3" >> /etc/hosts</pre><br />
<br />
Copy keys to servers where we need passwordless login:<br />
<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' <br />
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre> <br />
<br />
Setup shell:<br />
<pre>vi /root/.bashrc<br />
(add to bottom)<br />
alias h='history'<br />
alias vi='vim'<br />
alias j='jobs'<br />
export PS1="[\u@\h \w]# "<br />
alias dr='screen -dr'<br />
export EDITOR=vim<br />
export GREP_OPTIONS='--color=auto'<br />
export HISTFILESIZE=1000<br />
<br />
alias tip-switch-p20='cu -l ttyel00 -s 9600'<br />
alias tip-switch-p21='cu -l ttyel15 -s 9600'<br />
alias tip-switch-p22='cu -l ttyel14 -s 9600'<br />
alias tip-switch-p23='cu -l ttyel05 -s 9600'<br />
alias tip-switch-p24='cu -l ttyel06 -s 9600'<br />
alias tip-switch-p25='cu -l ttyel09 -s 9600'<br />
alias tip-switch-p26='cu -l ttyel07 -s 9600'<br />
alias tip-switch-p27='cu -l ttyel08 -s 9600'<br />
alias tip-firewall2='cu -l ttyel01 -s 115200'<br />
alias tip-nat2='cu -l /dev/ttyel02 -s 115200'<br />
alias tip-backup3='cu -l ttyel04 -s 38400'<br />
alias tip-bwdb2='cu -l ttyel03 -s 115200'<br />
alias tip-backup4='cu -l ttyel13 -s 115200'<br />
alias tip-jail3='cu -l ttyel11 -s 115200'<br />
<br />
Load new shell:<br />
source /root/.bashrc<br />
<br />
Setup snmpd (this is only valid for a server at castle):<br />
echo 'rocommunity jcread 10.1.4.5<br />
rocommunity jcread 10.1.4.3<br />
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf<br />
<br />
to see which iface it is, on backup2:<br />
<br />
snmpwalk -v 1 -c jcread 10.1.4.8 interface<br />
<br />
=== nfs ===<br />
<br />
Allow mounts from private net:<br />
echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports<br />
<br />
Restart nfsd:<br />
/etc/init.d/nfs-kernel-server restart<br />
<br />
=== bb ===<br />
<br />
Add user, group:<br />
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd<br />
echo "bb:x:1984:" >> /etc/group<br />
pwconv<br />
<br />
Create home:<br />
mkdir /home/bb<br />
chown bb.bb /home/bb<br />
cd ~bb<br />
<br />
Copy over and install files:<br />
<pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar .<br />
tar xf bb-linux.tar<br />
cd /home/bb/bbc1.9e-btf/etc</pre><br />
<br />
Configure main bb server:<br />
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts<br />
echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts<br />
<br />
Configure low disk alerts:<br />
<pre>echo "/:90:95<br />
/var:90:95<br />
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab</pre><br />
<br />
vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh<br />
(remove all | SORT xxxx since SORT is broken)<br />
<br />
chmod +r /var/log/messages<br />
<br />
./bbchkcfg.sh <br />
(y to questions)<br />
./bbchkhosts.sh<br />
(ignore ssh errors)<br />
<pre>cd ../..<br />
chown -R bb .<br />
su bb<br />
cd<br />
cd bbc1.9e-btf/src</pre><br />
<br />
<pre>make; make install<br />
cd ..<br />
./runbb.sh start<br />
more BBOUT</pre> <br />
(look for errors)<br />
exit<br />
<br />
<pre>vi /etc/rc.local<br />
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"<br />
</pre><br />
(before the exit 0)<br />
<br />
echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd<br />
<br />
Add f/w rule:<br />
ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984<br />
<br />
<pre>vi ~bb/bbc1.9e-btf/etc/bbdef-client.sh<br />
DFWARN=199<br />
DFPANIC=199</pre><br />
<br />
<br />
=== raid check ===<br />
<br />
==== 3ware ====<br />
<pre><br />
scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli<br />
scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh<br />
scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk<br />
vi /usr/local/sbin/checkraid.sh<br />
:%s/c0/c2/g<br />
<br />
crontab -e<br />
0 0 * * * /usr/local/sbin/3wraidchk</pre><br />
<br />
==== areca ====<br />
<pre><br />
cd /tmp<br />
wget http://www.areca.us/support/s_linux/cli/linuxcli_V1.10.0_120815.zip<br />
unzip linuxcli_V1.10.0_120815.zip<br />
cp linuxcli_V1.10.0_120815/x86_64/cli64 /usr/local/sbin/<br />
chmod 0700 /usr/local/sbin/cli64<br />
cli64 rsf info<br />
</pre><br />
<br />
<pre>scp backup2:/data4/bin/arecaraidchk /usr/local/sbin<br />
scp backup1:/usr/local/sbin/Sendmail.pm /usr/local/sbin<br />
<br />
crontab -e<br />
0 0 * * * /usr/local/sbin/arecaraidchk<br />
</pre><br />
<br />
<pre>cat > /root/verify.sh<br />
cli64 vsf info<br />
cli64 rsf info<br />
cli64 disk info<br />
cli64 event info<br />
echo press enter when ready to run verify ; read x<br />
<br />
cli64 vsf check vol=1<br />
</pre><br />
<br />
=== misc binaries ===<br />
<br />
scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive<br />
vi /usr/local/sbin/snapshot_archive<br />
(remove entries)<br />
<br />
crontab -e<br />
35 4 * * * /usr/local/sbin/snapshot_archive<br />
<br />
scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave<br />
scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone<br />
<br />
Since installing /bin/mail requires all sorts of packages (lame) we write a simple one here...which can only email johncompanies.com addr's unless you add relaying for this host:<br />
<br />
<pre><br />
cat > /bin/mail<br />
#!/usr/bin/perl<br />
use strict;<br />
use warnings;<br />
<br />
use lib '/usr/local/sbin';<br />
use Sendmail qw(sendmail);<br />
<br />
my $sub = $ARGV[1];<br />
my $to = $ARGV[2];<br />
<br />
<br />
my %mail = (<br />
To => $to,<br />
From => $to,<br />
Subject => $sub,<br />
Message => '',<br />
smtp => 'mail.johncompanies.com'<br />
);<br />
sendmail(%mail) || print "Error: $Sendmail::error";<br />
<br />
</pre><br />
<br />
chmod 0700 /bin/mail<br />
<br />
=== mkbackup ===<br />
<br />
mkdir /data/customers<br />
<br />
<pre>cat > /usr/local/sbin/mkbackup<br />
#!/bin/sh<br />
<br />
if test $1; then<br />
cid=$1<br />
else<br />
echo "ERROR: Usage: mkbackup cid GB ip Terminating."<br />
exit<br />
fi<br />
<br />
if test $2; then<br />
gb=$2<br />
else<br />
echo "ERROR: Usage: mkbackup cid GB ip Terminating."<br />
exit<br />
fi<br />
<br />
if test $3; then<br />
ip=$3<br />
else<br />
echo "ERROR: Usage: mkbackup cid GB ip Terminating."<br />
exit<br />
fi<br />
<br />
<br />
if test -e /data/customers/${cid}-file; then<br />
echo "ERROR: /data/customers/${cid}-file exists"<br />
exit<br />
else<br />
echo "touch /data/customers/${cid}-file"<br />
touch /data/customers/${cid}-file<br />
count=`echo $gb|awk '{print $1*1000}'`<br />
echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"<br />
dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count<br />
echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"<br />
/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file<br />
fi<br />
<br />
if test -e /data/customers/$cid; then<br />
echo "ERROR: /data/customers/$cid exists"<br />
exit<br />
else<br />
echo "mkdir /data/customers/${cid}"<br />
mkdir /data/customers/${cid}<br />
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"<br />
mount -o loop /data/customers/${cid}-file /data/customers/$cid<br />
df -h /data/customers/$cid<br />
<br />
echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh<br />
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh<br />
echo "" >> /etc/nfs_backup_mounts.sh<br />
<br />
echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports<br />
/etc/init.d/nfs-kernel-server restart<br />
tail /var/log/messages<br />
fi</pre><br />
<br />
chmod 0700 /usr/local/sbin/mkbackup<br />
<br />
vi /etc/rc.local<br />
add:<br />
/etc/nfs_backup_mounts.sh<br />
<br />
=== samba ===<br />
<br />
apt-get install samba<br />
<br />
vi /etc/samba/smb.conf<br />
<br />
; comment out any mounts, add:<br />
<br />
<pre>[data]<br />
read only = yes<br />
locking = no<br />
path = /data/iso<br />
guest ok = yes</pre><br />
<br />
/etc/init.d/smbd restart<br />
<br />
mkdir /data/iso<br />
<br />
Bring over some stuff from backup2<br />
<br />
<pre>cd /data/iso<br />
scp backup2:/d2/iso/3wfirmware.iso .<br />
scp backup2:/d2/iso/MD5SUMS .<br />
scp backup2:/d2/iso/bootimg.iso .<br />
scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso .<br />
scp backup2:/d2/iso/win98bootcd.iso .<br />
scp backup2:/d2/iso/acronis_bootdisk.iso .<br />
scp backup2:/d2/iso/memtest86-3.2.iso .</pre><br />
<br />
=== Moving from one server to another ===<br />
<br />
Here are the steps you would take to move settings and data from one server to a new backup server:<br />
<br />
* rsync over all /data/customers (we do this cause if we didn't use *-file it would copy over the files AND the data in the mountpoint)<br />
rsync -av --progress --ignore-times *-file root@10.1.2.33:/data/customers/<br />
after umounting all the customers, copy over the (empty) directories separately:<br />
for f in `find . -type d`; do rsync -av $f root@69.55.229.25:/data/customers; done<br />
<br />
* copy mount script<br />
[root@backup3 /data/customers]# scp /etc/nfs_backup_mounts.sh root@69.55.229.25:/etc/nfs_backup_mounts.sh<br />
<br />
* copy rc.local<br />
[root@backup3 /data/customers]# scp /etc/rc.local root@69.55.229.25:/etc/rc.local<br />
<br />
* copy /etc/exports<br />
[root@backup3 /data/customers]# scp /etc/exports root@69.55.229.25:/etc/exports<br />
<br />
* edit /etc/hostname on both machines (set current to oldbackup3)<br />
<br />
* edit /etc/network/interfaces (swap IPs). <br />
<br />
* stop mounts from mounting on old and new servers so it doesnt start with reboot right away:<br />
chmod 000 /etc/nfs_backup_mounts.sh<br />
<br />
* reboot both servers @ same time<br />
<br />
* check everything out<br />
<br />
* run /etc/nfs_backup_mounts.sh on new server<br />
<br />
* if switch port changed update mrtg to reflect correct port pub nic is on (on p20):<br />
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg<br />
<br />
= backup4 =<br />
== Summary ==<br />
This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs. Only FreeBSD virt customers directly accesses this server to perform their backups. <br />
<br />
* Location: castle, cab 3-7<br />
* OS: FreeNAS 9.3 (FreeBSD 9.3)<br />
* Networking: Priv IP: 10.1.2.9/24 AND 10.1.7.9/24, Pub IPs: 69.55.230.6/24 <br />
* Hardware: JC-08014<br />
Intel S5000VSA Motherboard<br />
1 x Intel Xeon E5410 @ 2.33GHz CPU<br />
3ware 9690SA-8I RAID Card w BBU<br />
16GB RAM<br />
Dual power supply.<br />
* Drives: 7 TB (6 x 2TB) ZFS RAIDZ2 array running on JBOD<br />
1 128 GB SSD system drive and 6 drive SATA bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap.<br />
<br />
* GUI management at http://backup4.johncompanies.com<br />
<br />
== Services provided ==<br />
* backup via rsync and nfs<br />
* samba<br />
* nfs<br />
* snmp?<br />
* bigbrother?<br />
<br />
== Usage ==<br />
* all data is stored under /data<br />
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt> ??<br />
* this server serves as an archive for exported db data from bwdb and old flow files. ??<br />
* customers are nfs-moutned under /data/users (/mnt/zfs/users) as zfs ?<br />
<br />
== management scripts ==<br />
* mkbackups?<br />
<br />
mkbackup <cid> GB <ip><br />
<br />
== Cronjobs ==<br />
0 0 * * * /usr/local/sbin/arecaraidchk <br />
RAID checks ?<br />
<br />
35 4 * * * /usr/local/sbin/snapshot_archive<br />
Rotate daily snapshots for infrastructure machine backups<br />
<br />
00 15 * * * /usr/local/sbin/snapshot_rotate<br />
Rotate daily snapshots for customer machine backups<br />
<br />
== Regular maintenance ==<br />
*[[Routine_Maintenance#A|Check on RAID health]]<br />
<br />
== Build ==<br />
<br />
= console = <br />
<br />
== Summary ==<br />
This box's only purpose is to serve as a means to connect to the digi serial multiplexer boxes at castle. Connect to it using the blue (cisco) ribbon cable with the beige RJ-45 to serial connector, 9600 8N1.<br />
<br />
* Location: castle, cab 3-8<br />
* OS: SunOS 5.8 (solaris)<br />
* Networking: Priv IP: 10.1.4.4<br />
* Hardware: Sun Netra<br />
<br />
<br />
To connect to consoles, ssh in as user 'console' and use the <tt>tip</tt> command to connect to devices listed in <tt>/etc/remote</tt><br />
<br />
i.e.<br />
tip switch-p1<br />
tip jail1<br />
<br />
== Configuring digi/ports == <br />
<br />
=== /etc/remote ===<br />
This is where the configuration/mapping for ports and custom names which we use along with the tip command to connect to various ports on the digi switches.<br />
<br />
We have 2 digi's at castle we connect to:<br />
<br />
#3-7 10.1.4.10<br />
virt15:dv=/dev/dty/CO001s:br#38400:el=^C^S^Q^U^D:ie=%$:oe=^D:<br />
virt13:dv=/dev/dty/CO002s:br#115200:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:<br />
<br />
and <br />
<br />
#3-6 10.1.4.11<br />
jail4:dv=/dev/dty/CP001s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:<br />
jail16:dv=/dev/dty/CP002s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:<br />
<br />
The only things you need to edit are the first part (i.e. <tt>jail4</tt>) and the speed (i.e. <tt>9600</tt>). You can decipher which port on the digi each line corresponds to by the <tt>CP001s or CO001s</tt> (port 1 on digi1 and digi2), <tt>CP002s or CO002s</tt> (port 2 on digi1 and digi2)<br />
<br />
=== drpadmin ===<br />
The tool you use to configure a device to a digi box is drpadmin:<br />
<br />
<pre>bash-2.03$ su<br />
Password:<br />
# drpadmin<br />
<br />
Please select an option (a)dd (d)elete (s)how (r)eset (q)uit : s<br />
0 10.1.4.10 32 CO 771 never 1027<br />
1 10.1.4.11 32 CP 771 never 1027<br />
2 65.116.11.2 8 el 771 never 1027<br />
<br />
Please select an option (a)dd (d)elete (s)how (r)eset (q)uit :</pre><br />
<br />
Use those commands above to modify the devices available.<br />
<br />
== Switching IP/hostname ==<br />
<br />
Edit:<br />
/etc/defaultrouter<br />
/etc/hosts<br />
/etc/hostname.hme0<br />
/etc/nodename<br />
Maybe needed to run: # ifconfig hme0 10.1.4.4 up<br />
<br />
= devweb =<br />
<br />
We do web development on devweb.johncompanies.com<br />
<br />
Currently this is a jail running on jail17 / 69.55.230.8<br />
<br />
If the jail is restarted, you will need to manually restart the web service with:<br />
httpsdctl restart<br />
<br />
All website development work should be done here first. It works exactly like and is setup like our [[Management_System_/_Public_Website_/_Signup_/_Account_Manager|main site]].<br />
<br />
<br />
= firewall2 =<br />
<br />
== Summary ==<br />
<br />
This machine is the primary firewall for the entire network at i2b. firewall3 is a hot standby replacement for <br />
firewall2. Both firewall2 and firewall3 should not be connected at the same time since they use the same internal<br />
and external IP addresses.<br />
<br />
* Location: i2b, cab 6<br />
* OS: FreeBSD 6.4 x86<br />
* Networking: Priv IP: 10.1.2.2, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. TODO: describe NIC location/orientation<br />
<br />
* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: 73 GB (2 x 73GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.<br />
<br />
== Services Provided ==<br />
* firewall (ipfw)<br />
* bigbrother for customer machines<br />
<br />
== Firewall Rule Configuration ==<br />
<br />
See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.<br />
<br />
== Disaster Recovery ==<br />
<br />
TODO: need backup f/w and instructions on how to move cables.<br />
<br />
Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)<br />
<br />
TODO<br />
<br />
Here's the config on the live firewall:<br />
<br />
<pre>kern_securelevel_enable="NO"<br />
portmap_enable="NO"<br />
sendmail_enable="NO"<br />
usbd_enable="YES"<br />
gateway_enable="YES"<br />
<br />
xntpd_enable="YES"<br />
nfs_client_enable="YES"<br />
nfs_reserved_port_only="YES"<br />
inetd_enable="YES"<br />
inetd_flags="-wW -a 10.1.2.2"<br />
<br />
fsck_y_enable="YES"<br />
background_fsck="NO"<br />
<br />
defaultrouter="66.181.18.2"<br />
hostname="firewall2.johncompanies.com"<br />
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"<br />
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"<br />
ifconfig_bge1_alias0="inet 69.55.231.1 netmask 255.255.255.0"<br />
ifconfig_bge1_alias1="inet 65.50.228.1 netmask 255.255.255.0"<br />
ifconfig_bge1_alias2="inet 65.50.229.1 netmask 255.255.255.0"<br />
ifconfig_bge1_alias3="inet 65.50.230.1 netmask 255.255.255.0"<br />
ifconfig_bge1_alias4="inet 65.50.231.1 netmask 255.255.255.0"<br />
ifconfig_bge1_alias5="inet 65.50.232.1 netmask 255.255.255.0"<br />
ifconfig_bge1_alias6="inet 65.50.233.1 netmask 255.255.255.0"<br />
ifconfig_bge1_alias7="inet 65.50.234.1 netmask 255.255.255.0"<br />
ifconfig_bge1_alias8="inet 65.50.235.1 netmask 255.255.255.0"<br />
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"<br />
sshd_enable="YES"<br />
usbd_enable="YES"<br />
</pre><br />
<br />
<br />
== Cronjobs ==<br />
30 3 * * * /usr/local/etc/rsync.backup<br />
Backup to backup3<br />
<br />
0 0 1 * * /sbin/ipfw zero<br />
0 0 1 * * /sbin/ipfw del 3 4 5<br />
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap). <br />
<br />
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count<br />
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count<br />
Capture counts periodically<br />
<br />
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;<br />
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.<br />
<br />
*/5 * * * * /usr/local/sbin/lsiraidchk<br />
Checking the health of the RAID array<br />
<br />
<br />
== DOS attacks ==<br />
<br />
See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.<br />
<br />
<br />
== build ==<br />
<br />
<pre>partition map:<br />
/ 58g<br />
swap 4g<br />
/var 512m<br />
/tmp 512m<br />
/usr 5.5g<br />
<br />
4. edit /etc/make.conf <br />
echo "WITHOUT_X11=yes \<br />
KERNCONF=firewall2 \<br />
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf<br />
<br />
5. add settings to /boot/loader.conf and /boot.config<br />
<br />
echo "-Dh" >> /boot.config<br />
<br />
echo 'console="comconsole,vidconsole" \<br />
boot_multicons="YES" \<br />
boot_serial="YES" \<br />
comconsole_speed="115200"' >> /boot/loader.conf<br />
<br />
<br />
6. turn off all ttyv's except 0 and 1 in /etc/ttys<br />
also turn on ttyd0, change type to vt100:<br />
vi /etc/ttys<br />
<br />
ttyv2 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv3 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv4 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv5 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv6 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv7 "/usr/libexec/getty Pc" cons25 off secure<br />
# Serial terminals<br />
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.<br />
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure<br />
<br />
kill -1 1<br />
<br />
on console server:<br />
vi /etc/remote<br />
(rename port to jail8 depending on where and which digi plugged into)<br />
test serial console<br />
<br />
<br />
7. populate hosts<br />
echo "69.55.230.10 backup2" >> /etc/hosts<br />
echo "69.55.230.11 backup1" >> /etc/hosts<br />
echo "10.1.2.3 backup3" >> /etc/hosts<br />
<br />
8. put key in authorized_keys on backup3<br />
cd<br />
ssh-keygen -t dsa -b 1024 <br />
(default location, leave password blank)<br />
<br />
Punch a hole in firewall1 to allow traffic to backup servers @ castle:<br />
<br />
ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22<br />
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22<br />
<br />
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys' <br />
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' <br />
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys' <br />
<br />
confirm that you can ssh to backup3 and backup 2 without getting a login prompt<br />
<br />
ssh backup3 hostname<br />
<br />
ssh backup2 hostname<br />
<br />
ssh backup1 hostname<br />
<br />
<br />
10. edit root's path and login script:<br />
vi /root/.cshrc<br />
<br />
Change alias entries (add G):<br />
alias la ls -aG<br />
alias lf ls -FAG<br />
alias ll ls -lAG<br />
alias ls ls -AG<br />
alias mbm mb mount<br />
alias mbu mb umount<br />
<br />
and alter the prompt, set the following:<br />
set prompt = "`/bin/hostname -s` %/# "<br />
<br />
11. install cvsup<br />
cd /usr/ports/net/cvsup-without-gui <br />
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null<br />
<br />
12. get latest sources for this release:<br />
cd /usr/src <br />
echo "*default host=cvsup4.freebsd.org\<br />
*default base=/usr\<br />
*default prefix=/usr\<br />
*default release=cvs tag=RELENG_6_4\<br />
*default delete use-rel-suffix\<br />
*default compress\<br />
src-all" > sup<br />
<br />
cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null<br />
<br />
13. configure new kernel. <br />
<br />
cd /usr/src/sys/i386/conf <br />
scp backup2:/mnt/data4/build/freebsd/firewall2-6.4 ./firewall2<br />
<br />
15. build, install kernel and world<br />
<br />
cd /boot<br />
<br />
mv kernel kernel.GENERIC<br />
cd kernel.GENERIC<br />
cd /usr/src<br />
make buildkernel installkernel<br />
<br />
make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null<br />
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)<br />
make installworld <br />
(2450: 3min, supermicro: 1min, 2950: :34)<br />
mergemaster -i<br />
<br />
17. populate /etc/rc.conf with IPs and NFS settings<br />
vi /etc/rc.conf<br />
<br />
kern_securelevel_enable="NO"<br />
portmap_enable="NO"<br />
sendmail_enable="NO"<br />
usbd_enable="YES"<br />
gateway_enable="YES"<br />
<br />
xntpd_enable="YES"<br />
nfs_client_enable="YES"<br />
nfs_reserved_port_only="YES"<br />
inetd_enable="YES"<br />
inetd_flags="-wW -a 10.1.2.2"<br />
<br />
ifconfig_bce1="inet 10.1.2.2 netmask 255.255.255.0"<br />
fsck_y_enable="YES"<br />
background_fsck="NO"<br />
<br />
defaultrouter="66.181.18.2"<br />
hostname="firewall2.johncompanies.com"<br />
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"<br />
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"<br />
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"<br />
sshd_enable="YES"<br />
usbd_enable="YES"<br />
<br />
20. reboot. Confirm new kernel is loaded<br />
<br />
uname -a<br />
<br />
21. update ports:<br />
cd /usr/ports<br />
echo "*default host=cvsup4.FreeBSD.org\<br />
*default base=/usr\<br />
*default prefix=/usr\<br />
*default release=cvs tag=RELENG_6_4\<br />
*default delete use-rel-suffix\<br />
*default compress\<br />
ports-all tag=." > sup<br />
<br />
cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null<br />
<br />
22. Install raid mgmt tool<br />
<br />
# linux base<br />
cd /usr/ports/devel/libtool22<br />
make install base<br />
<br />
cd /usr/ports/emulators/linux_base-fc4<br />
make install clean<br />
<br />
#linux-megamgr-5.20<br />
cd /usr/ports/sysutils/linux-megamgr<br />
make install clean<br />
<br />
# megarc-1.51<br />
cd /usr/ports/sysutils/megarc<br />
make install clean<br />
<br />
Test:<br />
rehash; megarc -ldInfo -a0 -l0<br />
<br />
23. install rsync from ports<br />
cd /usr/ports/net/rsync<br />
make install clean<br />
<br />
choose default options<br />
<br />
25. install bb client<br />
adduser<br />
Username: bb<br />
Full name: bb<br />
Uid (Leave empty for default): 1984<br />
Login group [bb]:<br />
Login group is bb. Invite bb into other groups? []:<br />
Login class [default]:<br />
Shell (sh csh tcsh nologin) [sh]: <br />
Home directory [/home/bb]:<br />
Use password-based authentication? [yes]:<br />
Use an empty password? (yes/no) [no]:<br />
Use a random password? (yes/no) [no]: yes<br />
Lock out the account after creation? [no]:<br />
Username : bb<br />
Password : <random><br />
Full Name : bb<br />
Uid : 1984<br />
Class :<br />
Groups : bb<br />
Home : /home/bb<br />
Shell : /bin/sh<br />
Locked : no<br />
OK? (yes/no): yes<br />
<br />
cd /usr/home/bb<br />
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .<br />
tar xvf bb-freebsd.tar<br />
<br />
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:<br />
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \<br />
10.1.2.1 firewall2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts<br />
<br />
vi /home/bb/bbc1.9e-btf/ext/openfiles <br />
MACHINE="firewall2,johncompanies,com" # HAS TO BE IN A,B,C FORM<br />
<br />
cd /usr/home/bb/bbc1.9e-btf/etc<br />
./bbchkcfg.sh <br />
(y to questions)<br />
./bbchkhosts.sh<br />
(ignore ssh errors)<br />
cd ../..<br />
chown -R bb .<br />
su bb<br />
cd<br />
cd bbc1.9e-btf/src<br />
make; make install<br />
cd ..<br />
<br />
vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh<br />
$1 $TOPARGS > $BBTMP/TOP.$$<br />
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$<br />
<br />
./runbb.sh start<br />
more BBOUT <br />
(look for errors)<br />
exit<br />
<br />
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh<br />
chmod +x /usr/local/etc/rc.d/bb.sh<br />
<br />
Punch a hole in the firewall to allow it to communicate with bb monitor:<br />
<br />
ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2<br />
<br />
<br />
27. configure bb on mail:<br />
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts<br />
66.181.18.3 firewall2.johncompanies.com # ssh<br />
<br />
su bb<br />
cd<br />
bbsrc/bb/runbb.sh restart ; exit<br />
<br />
29. configure ntp<br />
echo "server 10.1.2.1" > /etc/ntp.conf<br />
<br />
/usr/sbin/ntpd -p /var/run/ntpd.pid<br />
sleep 2; ntpq -p<br />
(confirm it’s able to reach our time server)<br />
<br />
echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh<br />
chmod 0700 /usr/local/etc/rc.d/ntp.sh<br />
<br />
30. fwd and reverse lookups on ns1c<br />
vr johncompanies.com<br />
(edit the PTR too)<br />
<br />
<br />
33. setup backups<br />
echo '#\!/bin/sh\<br />
backupdir=/data/firewall2/current\<br />
\<br />
## ENTRY /etc ' > /usr/local/etc/backup.config<br />
<br />
on backup3:<br />
setup backup dirs:<br />
ssh backup3 mkdir -p /data/firewall2/current<br />
<br />
on backup3, add the system to <br />
vi /usr/local/sbin/snapshot_archive<br />
<br />
scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup <br />
vi /usr/local/etc/rsync.backup <br />
backup1 > backup3<br />
<br />
crontab -e<br />
1 0 * * * /usr/local/etc/rsync.backup<br />
<br />
<br />
34. mkdir /root/logs<br />
<br />
35. edit sshd_config for security<br />
vi /etc/ssh/sshd_config<br />
ListenAddress 66.181.18.3<br />
ListenAddress 10.1.2.1<br />
<br />
kill -1 `cat /var/run/sshd.pid`<br />
<br />
35. raid chk<br />
<br />
cat > /usr/local/sbin/lsiraidchk<br />
#!/usr/bin/perl<br />
<br />
my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;<br />
<br />
foreach (@out) {<br />
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }<br />
#print $_;<br />
}<br />
<br />
36. add crontab entries<br />
crontab -e<br />
30 3 * * * /usr/local/etc/rsync.backup<br />
0 0 1 * * /sbin/ipfw zero<br />
0 0 1 * * /sbin/ipfw del 3 4 5<br />
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count<br />
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count<br />
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl; <br />
*/5 * * * * /usr/local/sbin/lsiraidchk<br />
<br />
#10 0 * * * rm /var/spool/clientmqueue/*<br />
<br />
scp /etc/makefwrules.pl user@64.163.14.48:~<br />
scp /etc/makepiperules.pl user@64.163.14.48:~<br />
mv /home/user/makefwrules.pl /etc<br />
mv /home/user/makepiperules.pl /etc<br />
touch /etc/firewall.sh<br />
mkdir /etc/oldrules/<br />
<br />
other binaries<br />
<br />
scp /usr/local/bin/rulemaker user@64.163.14.48:~<br />
mv ~user/rulemaker /usr/local/sbin<br />
scp ~user/Sendmail.pm user@64.163.14.48:~<br />
scp ~user/doswatch.pl user@64.163.14.48:~<br />
<br />
Setup basic ruleset<br />
<br />
ipfw add 00009 count udp from any to any<br />
ipfw add 00010 allow tcp from any to any established<br />
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss<br />
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18<br />
ipfw add 00012 deny tcp from any to any tcpflags syn,fin<br />
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg<br />
ipfw add 00012 allow icmp from any to any<br />
ipfw add 00014 deny tcp from any to any dst-port 135<br />
ipfw add 00150 skipto 65535 ip from any to any via em1 in<br />
<br />
IPKVM3:<br />
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139<br />
00098 deny ip from any to 69.55.230.10 dst-port 139<br />
</pre><br />
<br />
<br />
= firewall3 =<br />
<br />
== Summary ==<br />
<br />
This machine is the backup firewall for the network at i2b. <br />
<br />
* Location: i2b, cab ?<br />
* OS: FreeBSD 9.1 amd64<br />
* Networking: Priv IP: 10.1.2.5, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. <br />
<br />
<pre><br />
<br />
The internal network NIC is the left one on the motherboard (69.55.229.1/24, ...).<br />
The external network NIC is the right one on the motherboard (66.181.18.3/28).<br />
The PCI ethernet card is connected to our private network (10.1.2.5/24).<br />
<br />
</pre><br />
<br />
* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. <br />
* Drives: 160 GB (2 x 160GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.<br />
<br />
== Services Provided ==<br />
* firewall (ipfw)<br />
* bigbrother<br />
<br />
== Firewall Rule Configuration ==<br />
<br />
See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.<br />
<br />
== Disaster Recovery ==<br />
<br />
'''To put the backup firewall3 into service:'''<br />
<br />
<pre><br />
Move the internal cable (to our networks) from firewall2 to em1 which is the left most ethernet port (69.55.229.1).<br />
Move the external cable (to outside world) from firewall2 to em0 which is the port to the right on the motherboard (66.181.18.3).<br />
The PCI ethernet port (fxp0) should already be connected to private network (10.1.2.5).<br />
</pre><br />
<br />
Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)<br />
<br />
<pre><br />
kern_securelevel_enable="NO"<br />
portmap_enable="NO"<br />
sendmail_enable="NO"<br />
usbd_enable="YES"<br />
gateway_enable="YES"<br />
<br />
xntpd_enable="YES"<br />
nfs_client_enable="YES"<br />
nfs_reserved_port_only="YES"<br />
inetd_enable="YES"<br />
inetd_flags="-wW -a 10.1.2.5"<br />
<br />
fsck_y_enable="YES"<br />
background_fsck="NO"<br />
<br />
defaultrouter="66.181.18.2"<br />
hostname="firewall3.johncompanies.com"<br />
ifconfig_em0="inet 66.181.18.3 netmask 255.255.255.224"<br />
<br />
ifconfig_em1="inet 69.55.229.1 netmask 255.255.255.0"<br />
ifconfig_em1_alias0="inet 69.55.231.1 netmask 255.255.255.0"<br />
<br />
# ifconfig_em1_alias1="inet 65.50.228.1 netmask 255.255.255.0"<br />
# ifconfig_em1_alias2="inet 65.50.229.1 netmask 255.255.255.0"<br />
# ifconfig_em1_alias3="inet 65.50.230.1 netmask 255.255.255.0"<br />
# ifconfig_em1_alias4="inet 65.50.231.1 netmask 255.255.255.0"<br />
# ifconfig_em1_alias5="inet 65.50.232.1 netmask 255.255.255.0"<br />
# ifconfig_em1_alias6="inet 65.50.233.1 netmask 255.255.255.0"<br />
# ifconfig_em1_alias7="inet 65.50.234.1 netmask 255.255.255.0"<br />
# ifconfig_em1_alias8="inet 65.50.235.1 netmask 255.255.255.0"<br />
<br />
ifconfig_fxp0="inet 10.1.2.5 netmask 255.255.255.0"<br />
<br />
sshd_enable="YES"<br />
usbd_enable="YES"<br />
</pre><br />
<br />
== Cronjobs ==<br />
30 3 * * * /usr/local/etc/rsync.backup<br />
Backup to backup3<br />
<br />
0 0 1 * * /sbin/ipfw zero<br />
0 0 1 * * /sbin/ipfw del 3 4 5<br />
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap). <br />
<br />
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count<br />
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count<br />
Capture counts periodically<br />
<br />
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;<br />
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.<br />
<br />
*/5 * * * * /usr/local/sbin/lsiraidchk<br />
Checking the health of the RAID array<br />
<br />
<br />
== DOS attacks ==<br />
<br />
See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.<br />
<br />
<br />
== build ==<br />
<br />
<pre>partition map:<br />
/ 58g<br />
swap 4g<br />
/var 512m<br />
/tmp 512m<br />
/usr 5.5g<br />
<br />
4. edit /etc/make.conf <br />
echo "WITHOUT_X11=yes \<br />
KERNCONF=firewall3 \<br />
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf<br />
<br />
5. add settings to /boot/loader.conf and /boot.config<br />
<br />
echo "-Dh" >> /boot.config<br />
<br />
echo 'console="comconsole,vidconsole" \<br />
boot_multicons="YES" \<br />
boot_serial="YES" \<br />
comconsole_speed="115200"' >> /boot/loader.conf<br />
<br />
<br />
6. turn off all ttyv's except 0 and 1 in /etc/ttys<br />
also turn on ttyd0, change type to vt100:<br />
vi /etc/ttys<br />
<br />
ttyv2 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv3 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv4 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv5 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv6 "/usr/libexec/getty Pc" cons25 off secure<br />
ttyv7 "/usr/libexec/getty Pc" cons25 off secure<br />
# Serial terminals<br />
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.<br />
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure<br />
<br />
kill -1 1<br />
<br />
on console server:<br />
vi /etc/remote<br />
(rename port to jail8 depending on where and which digi plugged into)<br />
test serial console<br />
<br />
<br />
7. populate hosts<br />
echo "69.55.230.10 backup2" >> /etc/hosts<br />
echo "69.55.230.11 backup1" >> /etc/hosts<br />
echo "10.1.2.3 backup3" >> /etc/hosts<br />
<br />
8. put key in authorized_keys on backup3<br />
cd<br />
ssh-keygen -t dsa -b 1024 <br />
(default location, leave password blank)<br />
<br />
Punch a hole in firewall1 to allow traffic to backup servers @ castle:<br />
<br />
ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22<br />
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22<br />
<br />
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys' <br />
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' <br />
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys' <br />
<br />
confirm that you can ssh to backup3 and backup 2 without getting a login prompt<br />
<br />
ssh backup3 hostname<br />
<br />
ssh backup2 hostname<br />
<br />
ssh backup1 hostname<br />
<br />
<br />
10. edit root's path and login script:<br />
vi /root/.cshrc<br />
<br />
Change alias entries (add G):<br />
alias la ls -aG<br />
alias lf ls -FAG<br />
alias ll ls -lAG<br />
alias ls ls -AG<br />
alias mbm mb mount<br />
alias mbu mb umount<br />
<br />
and alter the prompt, set the following:<br />
set prompt = "`/bin/hostname -s` %/# "<br />
<br />
11. install cvsup<br />
cd /usr/ports/net/cvsup-without-gui <br />
make install clean; rehash; mail -s 'cvs installed' 8583619553@vtext.com < /dev/null<br />
<br />
12. get latest sources for this release:<br />
cd /usr/src <br />
echo "*default host=cvsup4.freebsd.org\<br />
*default base=/usr\<br />
*default prefix=/usr\<br />
*default release=cvs tag=RELENG_9_1\<br />
*default delete use-rel-suffix\<br />
*default compress\<br />
src-all" > sup<br />
<br />
cvsup sup ; mail -s 'cvs sup done' 8583619553@vtext.com < /dev/null<br />
<br />
13. configure new kernel. <br />
<br />
cd /usr/src/sys/amd64/conf <br />
scp backup2:/mnt/data4/build/freebsd/firewall3-9.1 ./firewall3<br />
<br />
15. build, install kernel and world<br />
<br />
cd /boot<br />
<br />
mv kernel kernel.GENERIC<br />
cd kernel.GENERIC<br />
cd /usr/src<br />
make buildkernel installkernel<br />
<br />
make buildworld ; mail -s 'buildworld done' 8583619553@vtext.com < /dev/null<br />
(supermicro: 2:15 mins, 2950: 38? mins)<br />
make installworld <br />
(2450: 3min, supermicro: 1min, 2950: :34)<br />
mergemaster -i<br />
<br />
17. populate /etc/rc.conf with IPs and NFS settings<br />
vi /etc/rc.conf<br />
<br />
kern_securelevel_enable="NO"<br />
portmap_enable="NO"<br />
sendmail_enable="NO"<br />
usbd_enable="YES"<br />
gateway_enable="YES"<br />
<br />
xntpd_enable="YES"<br />
nfs_client_enable="YES"<br />
nfs_reserved_port_only="YES"<br />
<br />
fsck_y_enable="YES"<br />
background_fsck="NO"<br />
<br />
hostname="firewall3.johncompanies.com"<br />
# external network<br />
ifconfig_em0="inet 66.181.18.3 netmask 255.255.255.224"<br />
<br />
# internal network<br />
ifconfig_em1="inet 69.55.229.1 netmask 255.255.255.0"<br />
ifconfig_em1_alias0="inet 69.55.231.1 netmask 255.255.255.0"<br />
<br />
ifconfig_em1_alias1="inet 65.50.228.1 netmask 255.255.255.0"<br />
ifconfig_em1_alias2="inet 65.50.229.1 netmask 255.255.255.0"<br />
ifconfig_em1_alias3="inet 65.50.230.1 netmask 255.255.255.0"<br />
ifconfig_em1_alias4="inet 65.50.231.1 netmask 255.255.255.0"<br />
ifconfig_em1_alias5="inet 65.50.232.1 netmask 255.255.255.0"<br />
ifconfig_em1_alias6="inet 65.50.233.1 netmask 255.255.255.0"<br />
ifconfig_em1_alias7="inet 65.50.234.1 netmask 255.255.255.0"<br />
ifconfig_em1_alias8="inet 65.50.235.1 netmask 255.255.255.0"<br />
<br />
defaultrouter="66.181.18.2"<br />
<br />
# private network<br />
ifconfig_fxp0="inet 10.1.2.5 netmask 255.255.255.0"<br />
<br />
inetd_enable="YES"<br />
inetd_flags="-wW -a 10.1.2.5"<br />
<br />
sshd_enable="YES"<br />
usbd_enable="YES"<br />
ntpd_enable="YES"<br />
# powerd_enable="YES"<br />
<br />
<br />
20. reboot. Confirm new kernel is loaded<br />
<br />
uname -a<br />
<br />
21. update ports:<br />
cd /usr/ports<br />
<br />
echo "*default host=cvsup4.FreeBSD.org\<br />
*default base=/usr\<br />
*default prefix=/usr\<br />
*default release=cvs tag=RELENG_9_1\<br />
*default delete use-rel-suffix\<br />
*default compress\<br />
ports-all tag=." > sup<br />
<br />
cvsup sup; mail -s 'cvs sup ports done' 8583619553@vtext.com < /dev/null<br />
<br />
22. Install raid mgmt tool<br />
<br />
# linux base<br />
cd /usr/ports/devel/libtool22<br />
make install base<br />
<br />
cd /usr/ports/emulators/linux_base-fc4<br />
make install clean<br />
<br />
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin<br />
cd /usr/local/sbin <br />
tar xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz<br />
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz<br />
<br />
23. install rsync from ports<br />
cd /usr/ports/net/rsync<br />
make install clean<br />
<br />
choose default options<br />
<br />
25. install bb client<br />
adduser<br />
Username: bb<br />
Full name: bb<br />
Uid (Leave empty for default): 1984<br />
Login group [bb]:<br />
Login group is bb. Invite bb into other groups? []:<br />
Login class [default]:<br />
Shell (sh csh tcsh nologin) [sh]: <br />
Home directory [/home/bb]:<br />
Use password-based authentication? [yes]:<br />
Use an empty password? (yes/no) [no]:<br />
Use a random password? (yes/no) [no]: yes<br />
Lock out the account after creation? [no]:<br />
Username : bb<br />
Password : <random><br />
Full Name : bb<br />
Uid : 1984<br />
Class :<br />
Groups : bb<br />
Home : /home/bb<br />
Shell : /bin/sh<br />
Locked : no<br />
OK? (yes/no): yes<br />
<br />
cd /usr/home/bb<br />
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .<br />
tar xvf bb-freebsd.tar<br />
<br />
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:<br />
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \<br />
10.1.2.5 firewall3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts<br />
<br />
vi /home/bb/bbc1.9e-btf/ext/openfiles <br />
MACHINE="firewall3,johncompanies,com" # HAS TO BE IN A,B,C FORM<br />
<br />
cd /usr/home/bb/bbc1.9e-btf/etc<br />
./bbchkcfg.sh <br />
(y to questions)<br />
./bbchkhosts.sh<br />
(ignore ssh errors)<br />
cd ../..<br />
chown -R bb .<br />
su bb<br />
cd<br />
cd bbc1.9e-btf/src<br />
make; make install<br />
cd ..<br />
<br />
vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh<br />
$1 $TOPARGS > $BBTMP/TOP.$$<br />
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$<br />
<br />
./runbb.sh start<br />
more BBOUT <br />
(look for errors)<br />
exit<br />
<br />
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh<br />
chmod +x /usr/local/etc/rc.d/bb.sh<br />
<br />
Punch a hole in the firewall to allow it to communicate with bb monitor:<br />
<br />
ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2<br />
<br />
<br />
27. configure bb on mail:<br />
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts<br />
66.181.18.3 firewall3.johncompanies.com # ssh<br />
<br />
su bb<br />
cd<br />
bbsrc/bb/runbb.sh restart ; exit<br />
<br />
29. configure ntp<br />
echo "server 10.1.2.1" > /etc/ntp.conf<br />
<br />
/usr/sbin/ntpd -p /var/run/ntpd.pid<br />
sleep 2; ntpq -p<br />
(confirm it’s able to reach our time server)<br />
<br />
echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh<br />
chmod 0700 /usr/local/etc/rc.d/ntp.sh<br />
<br />
30. fwd and reverse lookups on ns1c<br />
vr johncompanies.com<br />
(edit the PTR too)<br />
<br />
<br />
33. setup backups<br />
echo '#\!/bin/sh\<br />
backupdir=/data/firewall2/current\<br />
\<br />
## ENTRY /etc ' > /usr/local/etc/backup.config<br />
<br />
on backup3:<br />
setup backup dirs:<br />
ssh backup3 mkdir -p /data/firewall2/current<br />
<br />
on backup3, add the system to <br />
vi /usr/local/sbin/snapshot_archive<br />
<br />
scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup <br />
vi /usr/local/etc/rsync.backup <br />
backup1 > backup3<br />
<br />
crontab -e<br />
1 0 * * * /usr/local/etc/rsync.backup<br />
<br />
<br />
34. mkdir /root/logs<br />
<br />
35. edit sshd_config for security<br />
vi /etc/ssh/sshd_config<br />
ListenAddress 66.181.18.3<br />
ListenAddress 10.1.2.5<br />
<br />
kill -1 `cat /var/run/sshd.pid`<br />
<br />
35. raid chk<br />
<br />
cat > /usr/local/sbin/lsiraidchk<br />
#!/usr/bin/perl<br />
<br />
my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;<br />
<br />
foreach (@out) {<br />
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }<br />
#print $_;<br />
}<br />
<br />
36. add crontab entries<br />
crontab -e<br />
30 3 * * * /usr/local/etc/rsync.backup<br />
0 0 1 * * /sbin/ipfw zero<br />
0 0 1 * * /sbin/ipfw del 3 4 5<br />
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count<br />
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count<br />
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl; <br />
*/5 * * * * /usr/local/sbin/lsiraidchk<br />
<br />
#10 0 * * * rm /var/spool/clientmqueue/*<br />
<br />
scp /etc/makefwrules.pl user@64.163.14.48:~<br />
scp /etc/makepiperules.pl user@64.163.14.48:~<br />
mv /home/user/makefwrules.pl /etc<br />
mv /home/user/makepiperules.pl /etc<br />
touch /etc/firewall.sh<br />
mkdir /etc/oldrules/<br />
<br />
other binaries<br />
<br />
scp /usr/local/bin/rulemaker user@64.163.14.48:~<br />
mv ~user/rulemaker /usr/local/sbin<br />
scp ~user/Sendmail.pm user@64.163.14.48:~<br />
scp ~user/doswatch.pl user@64.163.14.48:~<br />
<br />
Setup basic ruleset<br />
<br />
ipfw add 00009 count udp from any to any<br />
ipfw add 00010 allow tcp from any to any established<br />
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss<br />
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18<br />
ipfw add 00012 deny tcp from any to any tcpflags syn,fin<br />
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg<br />
ipfw add 00012 allow icmp from any to any<br />
ipfw add 00014 deny tcp from any to any dst-port 135<br />
ipfw add 00150 skipto 65535 ip from any to any via em1 in<br />
<br />
IPKVM3:<br />
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139<br />
00098 deny ip from any to 69.55.230.10 dst-port 139<br />
</pre><br />
<br />
= wiki =<br />
<br />
The wiki (mediawiki) runs on nat2 in a jail running off 69.55.229.8<br />
<br />
The backup wiki lives on virt13 in CT 5 / 69.55.230.18<br />
<br />
== Setup jail ==<br />
<br />
<pre><br />
mkdir /mnt/data1/wiki-dir<br />
cd /usr/src<br />
make installworld DESTDIR=/mnt/data1/wiki-dir<br />
cd etc<br />
make distribution DESTDIR=/mnt/data1/wiki-dir<br />
<br />
mount -t devfs devfs /mnt/data1/wiki-dir/dev<br />
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset <br />
<br />
cd /mnt/data1/wiki-dir<br />
<br />
ln -sf dev/null kernel<br />
<br />
scp jail9:/usr/local/sbin/jkill /mnt/data1/wiki-dir/sbin<br />
<br />
jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh<br />
csh<br />
<br />
touch /etc/fstab<br />
echo 'network_interfaces=""\<br />
hostname="wiki.johncompanies.com"\<br />
kern_securelevel_enable="NO"\<br />
sendmail_enable="YES"\<br />
sshd_enable="YES"' > /etc/rc.conf<br />
<br />
echo "nameserver 69.55.229.3\<br />
nameserver 69.55.225.225" >> /etc/resolv.conf <br />
<br />
vi /etc/crontab<br />
<br />
(remove the adjkerntz lines )<br />
<br />
vi /etc/periodic/security/100.chksetuid<br />
<br />
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`<br />
with: MP='/' (use single quotes) <br />
<br />
mkdir -p /usr/compat/linux/dev<br />
<br />
adduser<br />
<br />
Username: user<br />
Full name: user<br />
Uid (Leave empty for default):<br />
Login group [user]:<br />
Login group is user. Invite user into other groups? []: wheel<br />
Login class [default]:<br />
Shell (sh csh tcsh nologin) [sh]:<br />
Home directory [/home/user]:<br />
Home directory permissions (Leave empty for default):<br />
Use password-based authentication? [yes]:<br />
Use an empty password? (yes/no) [no]:<br />
Use a random password? (yes/no) [no]: y<br />
Lock out the account after creation? [no]:<br />
Username : user<br />
Password : <random><br />
Full Name : user<br />
Uid : 1001<br />
Class :<br />
Groups : user<br />
Home : /home/user<br />
Home Mode :<br />
Shell : /bin/sh<br />
Locked : no<br />
OK? (yes/no): y<br />
adduser: INFO: Successfully added (user) to the user database.<br />
adduser: INFO: Password for (user) is: 901gmYjO<br />
Add another user? (yes/no): n<br />
Goodbye! <br />
<br />
vi /usr/home/user/.profile <br />
TERM=vt100; export TERM<br />
<br />
tzsetup<br />
<br />
newaliases <br />
<br />
rm /sbin/halt /sbin/reboot<br />
ln /sbin/jkill /sbin/halt<br />
ln /sbin/jkill /sbin/reboot<br />
<br />
vi /etc/syslog.conf<br />
#*.err;kern.warning;auth.notice;mail.crit /dev/console<br />
*.err;kern.warning;auth.notice;mail.crit /var/log/messages <br />
<br />
exit<br />
exit<br />
<br />
cd libexec<br />
chflags noschg ld-elf32.so.1<br />
chflags noschg ld-elf.so.1<br />
mv ld-elf32.so.1 ld-elf32.so.1-orig<br />
ln ld-elf.so.1 ld-elf32.so.1<br />
chflags schg ld-elf.so.1<br />
chflags schg ld-elf32.so.1<br />
<br />
cp -r /usr/ports /mnt/data1/wiki-dir/usr <br />
<br />
cat > /usr/local/etc/rc.d/wiki.sh<br />
mount -t devfs devfs /mnt/data1/wiki-dir/dev/<br />
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset<br />
jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh /etc/rc<br />
<br />
chmod 0700 /usr/local/etc/rc.d/wiki.sh<br />
</pre><br />
<br />
== mediawiki setup ==<br />
<br />
<pre><br />
<br />
cd /usr/ports/net/rsync<br />
make install clean<br />
<br />
cd /usr/ports/distfiles/<br />
fetch http://downloads.mysql.com/archives/mysql-5.5/mysql-5.5.4-m3.tar.gz<br />
cd /usr/ports/databases/mysql55-server<br />
make install clean<br />
<br />
cd /usr/ports/distfiles/<br />
fetch http://downloads.php.net/johannes/php-5.3.2.tar.bz2<br />
cd /usr/ports/lang/php52<br />
make install clean<br />
(build apache module)<br />
<br />
cd /usr/ports/lang/php5-extensions<br />
make install clean<br />
<br />
cd /usr/ports/www/apache22<br />
make install clean<br />
<br />
cd /usr/local/www/<br />
fetch http://download.wikimedia.org/mediawiki/1.19/mediawiki-1.19.2.tar.gz<br />
tar xzf mediawiki-1.19.2.tar.gz<br />
mv mediawiki-1.19.2 wiki<br />
<br />
vi /usr/local/etc/apache22/httpd.conf<br />
DocumentRoot "/usr/local/www/"<br />
<br />
Include etc/apache22/extra/vhost-wiki.conf<br />
Listen 443<br />
<br />
<IfModule mod_php5.c><br />
<FilesMatch "\.ph(p3?|tml)$"><br />
SetHandler application/x-httpd-php<br />
</FilesMatch><br />
<FilesMatch "\.phps$"><br />
SetHandler application/x-httpd-php-source<br />
</FilesMatch><br />
# To re-enable php in user directories comment the following lines<br />
# (from <IfModule ...> to </IfModule>.) Do NOT set it to On as it<br />
# prevents .htaccess files from disabling it.<br />
<IfModule mod_userdir.c><br />
<Directory /home/*/public_html><br />
php_admin_value engine Off<br />
</Directory><br />
</IfModule><br />
</IfModule><br />
<br />
<br />
cat > /usr/local/etc/apache22/extra/vhost-wiki.conf <br />
<VirtualHost *:443><br />
ServerAdmin support@johncompanies.com<br />
<br />
DocumentRoot /usr/local/www/wiki<br />
# <Directory /><br />
# Options FollowSymLinks<br />
# AllowOverride None<br />
# Order deny,allow<br />
# </Directory><br />
<Directory /usr/local/www/wiki><br />
Options Indexes FollowSymLinks MultiViews<br />
Deny from all<br />
AllowOverride AuthConfig<br />
Order allow,deny<br />
DirectoryIndex index.php<br />
#Allow from 69.55.233.195<br />
#Allow from boody.dyndns.org<br />
</Directory><br />
<br />
ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/<br />
<Directory "/usr/lib/cgi-bin"><br />
AllowOverride None<br />
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch<br />
Order allow,deny<br />
Allow from all<br />
</Directory><br />
<br />
ErrorLog /var/log/httpd-error.log<br />
<br />
# Possible values include: debug, info, notice, warn, error, crit,<br />
# alert, emerg.<br />
LogLevel warn<br />
<br />
CustomLog /var/log/httpd-access.log combined<br />
<br />
Alias /doc/ "/usr/share/doc/"<br />
<Directory "/usr/share/doc/"><br />
Options Indexes MultiViews FollowSymLinks<br />
AllowOverride None<br />
Order deny,allow<br />
Deny from all<br />
Allow from 127.0.0.0/255.0.0.0 ::1/128<br />
</Directory><br />
<br />
SSLEngine on<br />
SSLCertificateFile /usr/local/etc/apache22/ssl/server.crt<br />
SSLCertificateKeyFile /usr/local/etc/apache22/ssl/server.key<br />
<br />
</VirtualHost><br />
<br />
mkdir ssl<br />
cd ssl<br />
<br />
openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key<br />
US<br />
CA<br />
San Diego<br />
johncompanies.com<br />
johncompanies.com<br />
wiki.johncompanies.com<br />
support@johncompanies.com<br />
<br />
cat > /usr/local/www/wiki/.htaccess<br />
AuthType Basic<br />
AuthUserFile /usr/local/etc/apache22/wiki.passwd<br />
AuthName wiki<br />
require valid-user<br />
satisfy any<br />
<br />
cd /usr/local/etc/apache22<br />
htpasswd -c wiki.passwd admin<br />
<br />
https://69.55.229.8/index.php<br />
<br />
use mysql (innodb)<br />
wiki name: JCWiki<br />
Support / (mail pass) / support@johncompanies.com<br />
<br />
cat > /usr/local/www/wiki/LocalSettings.php<br />
<br />
<?php<br />
# This file was automatically generated by the MediaWiki 1.19.2<br />
# installer. If you make manual changes, please keep track in case you<br />
# need to recreate them later.<br />
#<br />
# See includes/DefaultSettings.php for all configurable settings<br />
# and their default values, but don't forget to make changes in _this_<br />
# file, not there.<br />
#<br />
# Further documentation for configuration settings may be found at:<br />
# http://www.mediawiki.org/wiki/Manual:Configuration_settings<br />
<br />
# Protect against web entry<br />
if ( !defined( 'MEDIAWIKI' ) ) {<br />
exit;<br />
}<br />
<br />
## Uncomment this to disable output compression<br />
# $wgDisableOutputCompression = true;<br />
<br />
$wgSitename = "JCWiki";<br />
<br />
## The URL base path to the directory containing the wiki;<br />
## defaults for all runtime URL paths are based off of this.<br />
## For more information on customizing the URLs please see:<br />
## http://www.mediawiki.org/wiki/Manual:Short_URL<br />
$wgScriptPath = "";<br />
$wgScriptExtension = ".php";<br />
<br />
## The protocol and server name to use in fully-qualified URLs<br />
$wgServer = "https://69.55.229.8";<br />
<br />
## The relative URL path to the skins directory<br />
$wgStylePath = "$wgScriptPath/skins";<br />
<br />
## The relative URL path to the logo. Make sure you change this from the default,<br />
## or else you'll overwrite your logo when you upgrade!<br />
#$wgLogo = "$wgStylePath/common/images/wiki.png";<br />
$wgLogo = "$wgStylePath/common/images/jclogo.gif";<br />
<br />
## UPO means: this is also a user preference option<br />
<br />
$wgEnableEmail = true;<br />
$wgEnableUserEmail = true; # UPO<br />
<br />
$wgEmergencyContact = "apache@69.55.229.8";<br />
$wgPasswordSender = "apache@69.55.229.8";<br />
<br />
$wgEnotifUserTalk = false; # UPO<br />
$wgEnotifWatchlist = false; # UPO<br />
$wgEmailAuthentication = true;<br />
<br />
## Database settings<br />
$wgDBtype = "mysql";<br />
$wgDBserver = "localhost";<br />
$wgDBname = "my_wiki";<br />
$wgDBuser = "root";<br />
$wgDBpassword = "";<br />
<br />
# MySQL specific settings<br />
$wgDBprefix = "";<br />
<br />
# MySQL table options to use during installation or update<br />
$wgDBTableOptions = "ENGINE=InnoDB, DEFAULT CHARSET=binary";<br />
<br />
# Experimental charset support for MySQL 5.0.<br />
$wgDBmysql5 = false;<br />
<br />
## Shared memory settings<br />
$wgMainCacheType = CACHE_NONE;<br />
$wgMemCachedServers = array();<br />
<br />
## To enable image uploads, make sure the 'images' directory<br />
## is writable, then set this to true:<br />
$wgEnableUploads = false;<br />
#$wgUseImageMagick = true;<br />
#$wgImageMagickConvertCommand = "/usr/bin/convert";<br />
<br />
# InstantCommons allows wiki to use images from http://commons.wikimedia.org<br />
$wgUseInstantCommons = false;<br />
<br />
## If you use ImageMagick (or any other shell command) on a<br />
## Linux server, this will need to be set to the name of an<br />
## available UTF-8 locale<br />
$wgShellLocale = "en_US.utf8";<br />
<br />
## If you want to use image uploads under safe mode,<br />
## create the directories images/archive, images/thumb and<br />
## images/temp, and make them all writable. Then uncomment<br />
## this, if it's not already uncommented:<br />
#$wgHashedUploadDirectory = false;<br />
<br />
## Set $wgCacheDirectory to a writable directory on the web server<br />
## to make your wiki go slightly faster. The directory should not<br />
## be publically accessible from the web.<br />
#$wgCacheDirectory = "$IP/cache";<br />
<br />
# Site language code, should be one of the list in ./languages/Names.php<br />
$wgLanguageCode = "en";<br />
<br />
$wgSecretKey = "abc699ef26890b49b4055430f8ebbd25e84cce21a7e53aeaec4d4313af4c9739";<br />
<br />
# Site upgrade key. Must be set to a string (default provided) to turn on the<br />
# web installer while LocalSettings.php is in place<br />
$wgUpgradeKey = "3196710f4a7d7332";<br />
<br />
## Default skin: you can change the default skin. Use the internal symbolic<br />
## names, ie 'standard', 'nostalgia', 'cologneblue', 'monobook', 'vector':<br />
$wgDefaultSkin = "vector";<br />
<br />
## For attaching licensing metadata to pages, and displaying an<br />
## appropriate copyright notice / icon. GNU Free Documentation<br />
## License and Creative Commons licenses are supported so far.<br />
$wgRightsPage = ""; # Set to the title of a wiki page that describes your license/copyright<br />
$wgRightsUrl = "";<br />
$wgRightsText = "";<br />
$wgRightsIcon = "";<br />
<br />
# Path to the GNU diff3 utility. Used for conflict resolution.<br />
$wgDiff3 = "/usr/bin/diff3";<br />
<br />
# Query string length limit for ResourceLoader. You should only set this if<br />
# your web server has a query string length limit (then set it to that limit),<br />
# or if you have suhosin.get.max_value_length set in php.ini (then set it to<br />
# that value)<br />
$wgResourceLoaderMaxQueryLength = -1;<br />
<br />
<br />
<br />
# End of automatically generated settings.<br />
# Add more configuration options below.<br />
<br />
<br />
</pre><br />
<br />
== copy/backup wiki ==<br />
on main/primary wiki:<br />
<pre><br />
/usr/local/etc/rc.d/mysql-server stop<br />
ssh 69.55.230.18 "/etc/init.d/mysql stop"<br />
rsync -av /var/db/mysql/my_wiki/ 69.55.230.18:/var/lib/mysql/my_wiki/<br />
rsync -av /var/db/mysql/ib* 69.55.230.18:/var/lib/mysql/<br />
/usr/local/etc/rc.d/mysql-server start<br />
ssh 69.55.230.18 "/etc/init.d/mysql start"<br />
<br />
</pre></div>75.80.128.147https://wiki.jcihosting.com/index.php?title=Routine_Maintenance&diff=2207Routine Maintenance2020-02-25T18:38:08Z<p>75.80.128.147: /* Check NetHere */</p>
<hr />
<div>= Daily Tasks =<br />
<br />
== check load graphs == <br />
<br />
Click on the Load link in mgmt <br />
<br />
This screen shows you load levels on our servers and network traffic for critical machines (firewalls, backup servers).<br />
<br />
If you see load high or increasing <br />
<br />
FreeBSD: <br />
run [[VPS_Management#jtop|jtop]] (or [[VPS_Management#jt|jt]] > 7.x) and see if there are any runaway processes. Here are some examples of entries in top that are<br />
definitely runaway processes:<br />
<br />
<pre>79481 root 64 0 2256K 1056K CPU1 1 58:16 87.40% 87.40% nano<br />
50650 1000 64 0 1852K 1112K RUN 0 207.9H 84.08% 84.08% screen<br />
14829 www 2 0 39100K 31736K accept 0 104:24 46.54% 6.54% httpd<br />
42065 root 61 0 1300K 844K RUN 1 47.8H 91.36% 91.36% ee<br />
1328 www 56 0 18440K 10796K CPU1 0 64.4H 97.71% 97.71% httpd<br />
26251 user 57 0 6124K 1160K CPU1 1 82.9H 98.44% 98.44% screen<br />
89874 root 60 0 1352K 892K RUN 1 33.8H 65.82% 65.82% dialog<br />
38656 1000 64 0 3088K 2136K CPU0 0 806:13 97.95% 97.95% StutBot<br />
27630 root 64 0 1396K 972K RUN 1 76.8H 86.47% 86.47% ee</pre><br />
<br />
Linux:<br />
run [[VPS_Management#vwe|vwe]] to see which VPS’s have high loads. From there run <tt>[[VPS_Management#vp|vp]] <veid></tt> and/or <tt>[[VPS_Management#vt|vt]] <veid></tt> to see what's going on in that system.<br />
[[VPS_Management#vzstat|vzstat]] will also give you a nice picture of whats going on, systems with high numbers in the mlat column are likely culprits.<br />
<br />
examples of out of control procs:<br />
<br />
<pre>12183 nobody 16 0 4916 1348 1340 R 45.5 0.0 4249m httpd<br />
29266 #502 16 0 1852 796 792 R 22.5 0.0 1104m vim<br />
23860 #41 16 0 5472 5472 2076 R 98.9 0.2 31:41 python<br />
19227 bin 19 0 1688 716 652 R 99.9 0.0 321:08 wtrs_ui<br />
7762 apache 16 0 268 236 224 R 85.7 0.0 1010m ptrace<br />
4624 #501 20 0 4304 2400 2044 R 53.6 0.1 284:32 YoSucker<br />
20451 #506 20 0 1876 820 816 R 17.2 0.0 169:35 vim<br />
8834 #514 20 0 900 724 672 R 77.6 0.0 382:30 neostats<br />
31815 apache 14 0 3176 3176 1696 R 74.4 0.1 6:15 counter</pre><br />
<br />
Just kill -9 them and be done with it.<br />
<br />
Also, anytime you see `kmod` or `ptrace` - kill those immediaely no matter how much they are using - they are attempts to exploit the linux ptrace bug. They won't work, but they suck a lot of CPU...<br />
<br />
Also, any other processes that are at 90-100% cpu usage and have been running for any long period of time should be killed except for mysqld processes on FreeBSD. See above.<br />
<br />
However, there is an exception:<br />
<br />
if it is a mysqld, we don't want to kill their database. What you want to do is <tt>[[VPS_Management#jpid|jpid]] <pid></tt> to see who owns it, and then email them the paste containing the instructions for the nanny. Or you can simply do a <tt>kill -1 PID</tt> on the process to restart it.<br />
<br />
=== Load averages jump at night ===<br />
<br />
The load averages on the FreeBSD systems may jump up at night between 1 and 4 am - this is because the backups are running - if this is what is causing a jump in load, you will see processes like `rsync` in top eating a lot of CPU time.<br />
<br />
== check backups ==<br />
<br />
mgmt -> Motnroing -> Backups and make sure every machine was backed up the previous nite. <br />
Also look at df on backup1 and backup2 to make sure no disk is approaching full, though bb should warn us in advance. Please note - errors encountered when a backup script on any of the particular systems run will gnerate an email to support@johncompanies.com so you can know immediately the day after if the directory to be backed up has been moved or no longer exists. A paste exists for this to notify the customer of a non-existant file/dir.<br />
<br />
== check bb for warnings ==<br />
mgmt -> BigBrother<br />
<br />
Some events don't generate pages (on purpose). You will only see them by going to the bb main page.<br />
<br />
== check jail5 for crashed VPSs ==<br />
<br />
On jail5<br />
<br />
notrunning<br />
<br />
To restart a VPS <br />
<br />
vm restart col0XXXX<br />
<br />
== Check NetHere ==<br />
<br />
Check the NetHere servers. To get into the servers, login to<br />
admin-1.nethere.net and su - to root.<br />
<br />
=== Mail systems ===<br />
Check for possible SPAMMERs.<br />
<br />
==== Incoming ====<br />
Check mta-1 and mta-2 count of customer logins for possible<br />
customer SPAM compromises.<br />
<br />
login_count /logs/maillog | tail -30<br />
<br />
==== Outgoing ====<br />
<br />
Check outgoing queues on relay-1 and relay-2<br />
<br />
mail_count | tail -30<br />
<br />
To clean up outgoing queues of unwanted SPAM on relay-1 and relay-2.<br />
<br />
mail_cleanup [ <sender's domain/username/message id> ... ]<br />
<br />
To just remove emails from some senders.<br />
<br />
rmmails <sender's domain/username/message id> ...<br />
<br />
=== Nagios ===<br />
<br />
Check for other problems on NetHere servers<br />
<br />
https://nagios.nethere.net<br />
<br />
=== Cacti ===<br />
<br />
Check bandwidth usage on servers<br />
<br />
https://cacti.nethere.net<br />
<br />
= Monthly Tasks =<br />
== rotate pine sent mail (1st of month) ==<br />
On the 1st of the month, before any emails are sent out, quit out of pine, then log back in. Send mail from last month will be archived.<br />
If you mess up and do it on the 3rd (for example), you can go into the previous month's saved email and save emails from the current month into the <tt>sent-mail</tt> (current month) mailbox.<br />
<br />
== b/w caps ==<br />
On the 1st: remove any bwcaps put into the firewall (only really applies if a bwcap was added cause someone went over on b/w):<br />
<br />
ipfw list|grep pipe<br />
ipfw del [each rule listed]<br />
<br />
NOTE: this cronjob on newgateway will do some of that for you, provided you used one of the following pipe #s:<br />
0 0 1 * * /sbin/ipfw del 3 4 5 17331<br />
<br />
-------------------<br />
<br />
We really don’t do this anymore since we have centralized traffic accounting with netflow, but for posterity:<br />
<br />
Make sure all machines reset counters to 0 after midnight on the 1st<br />
Make sure they dumped a counter<br />
<br />
On each jail run:<br />
trafficgather.pl<br />
<br />
And on each virt:<br />
linuxtrafficgather.pl<br />
<br />
== Monthly RAID checks ==<br />
<br />
Every month we check the health of and verfy the parity on all our RAID-based systems.<br />
To facilitate this, we've created a simple script to start the process:<br />
<br />
sh /root/verify.sh<br />
<br />
=== Adaptec controllers ===<br />
Here's some sample output:<br />
<pre><br />
mail /usr/local/www/scripts# sh /root/verify.sh<br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> container list /f<br />
Executing: container list /full=TRUE<br />
Num Total Oth Chunk Scsi Partition<br />
Creation System<br />
Label Type Size Ctr Size Usage B:ID:L Offset:Size State RO Lk Task Done% Ent<br />
Date Time Files<br />
----- ------ ------ --- ------ ------- ------ ------------- ------- -- -- ------- ------ ---<br />
------ -------- ------<br />
0 Mirror 33.9GB Open 0:01:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:32<br />
/dev/aacd0 mirror0 0:00:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:32<br />
<br />
1 Mirror 33.9GB Open 0:02:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:50<br />
/dev/aacd1 mirror1 0:03:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:50<br />
<br />
<br />
AAC0> disk list /f<br />
Executing: disk list /full=TRUE<br />
<br />
B:ID:L Device Type Removable media Vendor-ID Product-ID Rev Blocks Bytes/Bl<br />
ock Usage Shared Rate<br />
------ -------------- --------------- --------- ---------------- ----- --------- --------<br />
--- ---------------- ------ ----<br />
0:00:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:01:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:02:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:03:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
<br />
AAC0> disk show smart<br />
Executing: disk show smart<br />
<br />
Smart Method of Enable<br />
Capable Informational Exception Performance Error<br />
B:ID:L Device Exceptions(MRIE) Control Enabled Count<br />
------ ------- ---------------- --------- ----------- ------<br />
0:00:0 Y 6 Y N 0<br />
0:01:0 Y 6 Y N 0<br />
0:02:0 Y 6 Y N 0<br />
0:03:0 Y 6 Y N 0<br />
0:06:0 N<br />
<br />
AAC0> task list<br />
Executing: task list<br />
<br />
Controller Tasks<br />
<br />
TaskId Function Done% Container State Specific1 Specific2<br />
------ -------- ------- --------- ----- --------- ---------<br />
<br />
No tasks currently running on controller<br />
<br />
AAC0> dia sh hi<br />
Executing: diagnostic show history<br />
No switches specified, defaulting to "/current".<br />
<br />
<br />
<br />
*** HISTORY BUFFER FROM CURRENT CONTROLLER RUN ***<br />
<br />
[00]: GetDiskLogEntry: container - 1, entry return 0<br />
[01]: Container 1 started SCRUB task<br />
[02]: Starting Mirror:1 scrub<br />
[03]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[04]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[05]: UpdateDiskLogIndex - Set - container 0, index 1<br />
[06]: GetDiskLogEntry: container - 0, entry return 1<br />
[07]: Container 0 started SCRUB task<br />
[08]: Starting Mirror:0 scrub<br />
[09]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[10]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[11]: Mirror Scrub Container:1 ErrorsFound:0<br />
[12]: Clear disk log: sector - 80, driveno 2<br />
[13]: Clear disk log: sector - 80, driveno 3<br />
[14]: Container 1 completed SCRUB task:<br />
[15]: Mirror Scrub Container:0 ErrorsFound:0<br />
[16]: Clear disk log: sector - 81, driveno 1<br />
[17]: Clear disk log: sector - 81, driveno 0<br />
[18]: Container 0 completed SCRUB task:<br />
[19]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[20]: GetDiskLogEntry: container - 0, entry return 0<br />
[21]: Container 0 started SCRUB task<br />
[22]: Starting Mirror:0 scrub<br />
[23]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[24]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[25]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[26]: GetDiskLogEntry: container - 1, entry return 1<br />
[27]: Container 1 started SCRUB task<br />
[28]: Starting Mirror:1 scrub<br />
[29]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[30]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[31]: Mirror Scrub Container:1 ErrorsFound:0<br />
[32]: Clear disk log: sector - 81, driveno 2<br />
[33]: Clear disk log: sector - 81, driveno 3<br />
[34]: Container 1 completed SCRUB task:<br />
[35]: Mirror Scrub Container:0 ErrorsFound:0<br />
[36]: Clear disk log: sector - 80, driveno 1<br />
[37]: Clear disk log: sector - 80, driveno 0<br />
[38]: Container 0 completed SCRUB task:<br />
[39]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[40]: GetDiskLogEntry: container - 0, entry return 0<br />
[41]: Container 0 started SCRUB task<br />
[42]: Starting Mirror:0 scrub<br />
[43]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[44]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[45]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[46]: GetDiskLogEntry: container - 1, entry return 1<br />
[47]: Container 1 started SCRUB task<br />
[48]: Starting Mirror:1 scrub<br />
[49]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[50]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[51]: Mirror Scrub Container:1 ErrorsFound:0<br />
[52]: Clear disk log: sector - 81, driveno 2<br />
[53]: Clear disk log: sector - 81, driveno 3<br />
[54]: Container 1 completed SCRUB task:<br />
[55]: Mirror Scrub Container:0 ErrorsFound:0<br />
[56]: Clear disk log: sector - 80, driveno 1<br />
[57]: Clear disk log: sector - 80, driveno 0<br />
[58]: Container 0 completed SCRUB task:<br />
[59]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[60]: GetDiskLogEntry: container - 0, entry return 0<br />
[61]: Container 0 started SCRUB task<br />
[62]: Starting Mirror:0 scrub<br />
[63]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[64]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[65]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[66]: GetDiskLogEntry: container - 1, entry return 1<br />
[67]: Container 1 started SCRUB task<br />
[68]: Starting Mirror:1 scrub<br />
[69]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[70]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[71]: Mirror Scrub Container:1 ErrorsFound:0<br />
[72]: Clear disk log: sector - 81, driveno 2<br />
[73]: Clear disk log: sector - 81, driveno 3<br />
[74]: Container 1 completed SCRUB task:<br />
[75]: Mirror Scrub Container:0 ErrorsFound:0<br />
[76]: Clear disk log: sector - 80, driveno 1<br />
[77]: Clear disk log: sector - 80, driveno 0<br />
[78]: Container 0 completed SCRUB task:<br />
[79]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[80]: GetDiskLogEntry: container - 0, entry return 0<br />
[81]: Container 0 started SCRUB task<br />
[82]: Starting Mirror:0 scrub<br />
[83]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[84]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[85]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[86]: GetDiskLogEntry: container - 1, entry return 1<br />
[87]: Container 1 started SCRUB task<br />
[88]: Starting Mirror:1 scrub<br />
[89]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[90]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[91]: Mirror Scrub Container:1 ErrorsFound:0<br />
[92]: Clear disk log: sector - 81, driveno 2<br />
[93]: Clear disk log: sector - 81, driveno 3<br />
[94]: Container 1 completed SCRUB task:<br />
[95]: Mirror Scrub Container:0 ErrorsFound:0<br />
[96]: Clear disk log: sector - 80, driveno 1<br />
[97]: Clear disk log: sector - 80, driveno 0<br />
[98]: Container 0 completed SCRUB task:<br />
[99]:<br />
<br />
========================<br />
History Output Complete.<br />
<br />
AAC0><br />
AAC0> exit<br />
Executing: exit<br />
<br />
press enter when ready to run verify <INS><br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> contai scr 0<br />
Executing: container scrub 0<br />
<br />
AAC0> contai scr 1<br />
Executing: container scrub 1<br />
<br />
AAC0> exit<br />
Executing: exit<br />
<br />
when done run: <br />
<br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
<br />
<br />
Nov 1 10:32:46 mail /kernel: aac0: **Monitor** Container 0 started SCRUB task<br />
Nov 1 10:32:47 mail /kernel: aac0: **Monitor** Container 1 started SCRUB task<br />
<br />
</pre><br />
<br />
Here's an analysis of what we're seeing and what we're looking for:<br />
<br />
<pre><br />
AAC0> container list /f<br />
Executing: container list /full=TRUE<br />
Num Total Oth Chunk Scsi Partition<br />
Creation System<br />
Label Type Size Ctr Size Usage B:ID:L Offset:Size State RO Lk Task Done% Ent<br />
Date Time Files<br />
----- ------ ------ --- ------ ------- ------ ------------- ------- -- -- ------- ------ ---<br />
------ -------- ------<br />
0 Mirror 33.9GB Open 0:01:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:32<br />
/dev/aacd0 mirror0 0:00:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:32<br />
<br />
1 Mirror 33.9GB Open 0:02:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:50<br />
/dev/aacd1 mirror1 0:03:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:50<br />
</pre><br />
<br />
This is showing you the health of the arrays. You're looking for ''Normal'' under the State column, and the absence of a ! in the sector size - sometimes, you'll see this:<br />
64.0KB!33.9GB <br />
<br />
That indicates a problem.<br />
<br />
<pre><br />
AAC0> disk show smart<br />
Executing: disk show smart<br />
<br />
Smart Method of Enable<br />
Capable Informational Exception Performance Error<br />
B:ID:L Device Exceptions(MRIE) Control Enabled Count<br />
------ ------- ---------------- --------- ----------- ------<br />
0:00:0 Y 6 Y N 0<br />
0:01:0 Y 6 Y N 0<br />
0:02:0 Y 6 Y N 0<br />
0:03:0 Y 6 Y N 0<br />
0:06:0 N<br />
</pre><br />
<br />
This shows you a SMART report output. Looking for values in the Error Count column.<br />
<pre><br />
AAC0> task list<br />
Executing: task list<br />
<br />
Controller Tasks<br />
<br />
TaskId Function Done% Container State Specific1 Specific2<br />
------ -------- ------- --------- ----- --------- ---------<br />
<br />
No tasks currently running on controller<br />
</pre><br />
<br />
Look for absence of tasks running- a bad thing would be to see a rebuild or verify running when you didn't initiate it.<br />
<br />
With the history output, you're looking for any anomalies or events since the last time a verify was run. If you see a drive with lots of problems, you may want to take backups before allowing the verify to run since it could replicate errors onto the good drive.<br />
<br />
After you see the history output, it will prompt you to press enter to run the verify. If you're happy with all the output you're seeing- mirror is healthy, history looks good, it's safe to proceed. Otherwise ^C to exit. After hitting enter it will start the verify and start to tail the messages log file (so you can easily see when the verify is complete). Here's what that'll look like:<br />
<br />
<pre><br />
Nov 1 14:38:08 mail /kernel: aac0: **Monitor** Container 1 completed SCRUB task:<br />
Nov 1 14:46:45 mail /kernel: aac0: **Monitor** Container 0 completed SCRUB task:<br />
</pre><br />
<br />
So, putting it all together, after hitting enter to start the verify, you'll see:<br />
<br />
<pre><br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> contai scr 0<br />
Executing: container scrub 0<br />
<br />
AAC0> contai scr 1<br />
Executing: container scrub 1<br />
<br />
AAC0> exit<br />
Executing: exit<br />
<br />
when done run: <br />
<br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
<br />
<br />
Nov 1 10:32:46 mail /kernel: aac0: **Monitor** Container 0 started SCRUB task<br />
Nov 1 10:32:47 mail /kernel: aac0: **Monitor** Container 1 started SCRUB task<br />
<br />
</pre><br />
<br />
When the scrub(s) (verify) are complete - if the server has multiple logical drives, it will run both in parallel - you should exit the tail of the log file (^C) and run:<br />
<br />
<pre><br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
</pre><br />
<br />
Which will show you the diagnostic history, you're looking for the results of the most recent scrub:<br />
<br />
<pre><br />
[100]: Mirror Scrub Container:1 ErrorsFound:0<br />
[101]: Clear disk log: sector - 81, driveno 2<br />
[102]: Clear disk log: sector - 81, driveno 3<br />
[103]: Container 1 completed SCRUB task:<br />
[104]: Mirror Scrub Container:0 ErrorsFound:0<br />
[105]: Clear disk log: sector - 80, driveno 1<br />
[106]: Clear disk log: sector - 80, driveno 0<br />
[107]: Container 0 completed SCRUB task:<br />
</pre><br />
<br />
^C to exit the RAID CLI.<br />
<br />
If you see:<br />
<br />
[104]: Mirror Scrub Container:0 ErrorsFound:5<br />
<br />
You'll want to rerun the verify on that drive till it shows 0, or perhaps replace the drive- you should be able to see from the output which drive had the problem.<br />
<br />
Depending on the size and how busy the drive is, the verify can take anywhere from an hour to the better part of a day.<br />
<br />
You will notice that the diagnostic history is not shown on our modern adaptec cards (i.e. any adaptec card not in a Dell 2450). The reason for this is the history is never cleared, so there's simply too much data to show and it just crashes the CLI. So, don't bother trying to see it...which does make it hard to see if there are problems going on, so you just need to watch the scrub to see it goes to 100%. You will also notice that on some servers there's no tail of messages. Again, this is cause no data is shown there about the completion of the scrub. The thing to do here is to go into the CLI and continue to <tt>show tasks</tt> to monitor scrub progress.<br />
<br />
See [[RAIC_CLI#Adaptec|Adaptec RAID CLI Reference]] for more details on how to use the CLI<br />
<br />
=== DELL (LSI-based) SAS controllers ===<br />
<br />
Here's what the output looks like when running verify.sh on a LSI-based card:<br />
<br />
<pre><br />
jail2 /mnt/data2# sh /root/verify.sh<br />
<br />
Adapter #0<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 0<br />
Device Id: 0<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018396142<br />
SAS Address(1): 0x0<br />
Connected Port Number: 0(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DESN<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 1<br />
Device Id: 1<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018395db2<br />
SAS Address(1): 0x0<br />
Connected Port Number: 1(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DERV<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 2<br />
Device Id: 2<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50006eece89<br />
SAS Address(1): 0x0<br />
Connected Port Number: 2(path0)<br />
Inquiry Data: SEAGATE ST3300555SS T2113LM4BFBZ<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 3<br />
Device Id: 3<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50006eee035<br />
SAS Address(1): 0x0<br />
Connected Port Number: 3(path0)<br />
Inquiry Data: SEAGATE ST3300555SS T2113LM4BGF7<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 4<br />
Device Id: 4<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50004bd7ea5<br />
SAS Address(1): 0x0<br />
Connected Port Number: 4(path0)<br />
Inquiry Data: SEAGATE ST3300656SS HS093QP0G8SW<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 5<br />
Device Id: 5<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e01f1c4112<br />
SAS Address(1): 0x0<br />
Connected Port Number: 5(path0)<br />
Inquiry Data: FUJITSU MBA3300RC D306BJ15P9201W06<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 0 (Target Id: 0)<br />
Name:<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:139392MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 1 (Target Id: 1)<br />
Name:MIRROR1<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 2 (Target Id: 2)<br />
Name:MIRROR2<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
Battery FRU : N/A<br />
Battery Warning : Enabled<br />
Memory Correctable Errors : 0<br />
Memory Uncorrectable Errors : 0<br />
BBU : Present<br />
BBU : Yes<br />
Cache When BBU Bad : Disabled<br />
press enter when ready to run verify<br />
<br />
</pre><br />
<br />
Before pressing enter, here's what we're looking for:<br />
<br />
<pre><br />
Enclosure Device ID: 32<br />
Slot Number: 0<br />
Device Id: 0<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018396142<br />
SAS Address(1): 0x0<br />
Connected Port Number: 0(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DESN<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
</pre><br />
<br />
This is the output shown for each physical drive in the system. We're looking to confirm it's Firmware state is ''Online'', and Media Error Count, Other Error Count, and Predictive Failure Count are all zero (or near zero).<br />
<br />
<pre><br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 1 (Target Id: 1)<br />
Name:MIRROR1<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
</pre><br />
<br />
This is the output for each logical drive. We're looking for State ''Optimal''. Also confirm <tt>Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU</tt><br />
<br />
<pre><br />
Exit Code: 0x00<br />
Battery FRU : N/A<br />
Battery Warning : Enabled<br />
Memory Correctable Errors : 0<br />
Memory Uncorrectable Errors : 0<br />
BBU : Present<br />
BBU : Yes<br />
Cache When BBU Bad : Disabled<br />
</pre><br />
<br />
Confirm that the battery is present and error-free.<br />
<br />
If all that checks out, you're ready to proceed with the verify. After pressing enter, the verify is started and here's what you see:<br />
<pre><br />
Start Check Consistency on Virtual Drive 0 (target id: 0) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Start Check Consistency on Virtual Drive 1 (target id: 1) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Start Check Consistency on Virtual Drive 2 (target id: 2) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Check Consistency<br />
<br />
Progress of Virtual Drives...<br />
<br />
Virtual Drive # Percent Complete Time Elps<br />
0 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:03<br />
1 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:02<br />
2 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:01<br />
<br />
Press <ESC> key to quit...<br />
</pre><br />
<br />
The progress for each drive is displayed until all drives have completed the verify. We just want to make sure that each drive goes to completion. No followup is needed...though there probably is a log or history where we can get more info.<br />
<br />
You will notice that jail7 does not run a verify- that's on purpose. The last time we tried this it crashed the system. So, this must be run from the BIOS (take the system offline for a couple hours).<br />
<br />
See [[RAIC_CLI#LSI|LSI RAID CLI Reference]] for more details on how to use the CLI<br />
<br />
=== LSI-based controllers (megaraid) ===<br />
<br />
There is a CLI for this however it's easier to do this with a curses GUI app: megaraid<br />
<br />
Currently only on these servers: virt15, virt16, and firewall2<br />
<br />
To run:<br />
<br />
# cd /usr/local/sbin/; megamgr<br />
<br />
Main menu:<br />
<pre>²ÚÄÄManagement MenuÄÄ¿²<br />
²³ Configure ³²<br />
²³ Initialize ³²<br />
²³ Objects ³²<br />
²³ Rebuild ³²<br />
²³ Check Consistency ³²<br />
²³ Advanced Menu ³²<br />
²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²</pre><br />
<br />
Before you check consistency, make sure the arrays are healthy.<br />
<br />
Objects -> Physical Drive<br />
<br />
Then look to make sure they're all <tt>ONLIN</tt><br />
<pre><br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄObjects - PHYSICAL DRIVE SELECTION MENUÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²<br />
²²²²²²²²³ ³²²²²²²<br />
²²²²²²²²³ Channel-1 ³²²²²²²<br />
²ÚÄÄMana³ ID ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»x ³²²²²²²<br />
²³ Confi³ 0º* ONLIN A01-01º ³²²²²²²<br />
²³ Initi³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²³ Objec³ 1º* ONLIN A01-02º ³²²²²²²<br />
²³ Rebui³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²³ Check³ 2º* ONLIN A02-01º ³²²²²²²<br />
²³ Advan³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²ÀÄÄÄÄÄij 3º* ONLIN A02-02º ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 4º* ONLIN A03-01º ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 5º* ONLIN A03-02ºþ ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 6º* º ³²²²²²²<br />
²²²²²²²²³ ÈÍÍÍÍÍÍÍÍÍÍÍÍÍͼx ³²²²²²²<br />
²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²<br />
²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²²²<br />
²²²²²²²²³Ch-1 ID-5 DISK 140013MB SEAGATE ST3146707LC 0003 ³²²²²²²²²<br />
²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²<br />
</pre><br />
<br />
<br />
Once that's done, hit escape once then the back arror to move back to the Objects menu.<br />
So you select Objects -> Logical Drive -> Logical Drive 1 -> Check Consistency -> YES<br />
<br />
<pre><br />
<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÚÄLogical Drives(02)Ä¿²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ Logical Drive 1 ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²ÚÄÄÄÄObjectsÄÄij Logical Drive 2 ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²ÚÄÄManagemen³ Adapter ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Configure ³ Logical Drive ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Initialize³ Physical Drive ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Objects ³ Channel ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Rebuild ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Check Consistency ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Advanced Menu ³²²²²²²²ÚÄÄÄÄLogical Drive 1ÄÄÄÄÄ¿²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²³ Initialize ÚÄCheck Consistency-1 ?Ä¿²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ Check Consiste³ YES ³²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ View/Update Pa³ NO ³²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³Select YES Or NO³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
</pre><br />
<br />
Then watch the progress. When done, escape back to Logical Drive then repeat for Logical Drive 2. If you ^C or accidentally escape out, you can come back in running the same commands and watch the progress again (it won't restart).<br />
<br />
You can exit megamgr by escaping out or ^C<br />
<br />
=== 3ware ===<br />
<br />
We are using 3ware controllers on backup1 & backup2. Running the verify script will give you different output based on the type of controller:<br />
<br />
<pre><br />
backup2 /d2# sh /root/verify.sh<br />
Controller: c0<br />
-------------<br />
Driver: 1.50.01.002<br />
Model: 7500-8<br />
FW: FE7X 1.05.00.068<br />
BIOS: BE7X 1.08.00.048<br />
Monitor: ME7X 1.01.00.040<br />
Serial #: F11605A3180172<br />
PCB: Rev3<br />
PCHIP: 1.30-33<br />
ACHIP: 3.20<br />
<br />
<br />
# of units: 3<br />
Unit 0: JBOD 186.31 GB ( 390721968 blocks): OK<br />
Unit 1: RAID 5 465.77 GB ( 976790016 blocks): DEGRADED<br />
Unit 5: RAID 5 698.65 GB ( 1465185024 blocks): DEGRADED<br />
<br />
# of ports: 8<br />
Port 0: WDC WD2000JB-00KFA0 WD-WCAMT1451690 186.31 GB (390721968 blocks): OK(unit 0)<br />
Port 1: WDC WD2500JB-00GVC0 WD-WCAL78219488 232.88 GB (488397168 blocks): OK(unit 1)<br />
Port 2: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT)<br />
Port 3: WDC WD2500JB-00GVC0 WD-WMAL73882417 232.88 GB (488397168 blocks): OK(unit 1)<br />
Port 4: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT)<br />
Port 5: WDC WD2500JB-00GVA0 WD-WMAL71338097 232.88 GB (488397168 blocks): OK(unit 5)<br />
Port 6: WDC WD2500JB-32EVA0 WD-WMAEH1301595 232.88 GB (488397168 blocks): OK(unit 5)<br />
Port 7: WDC WD2500JB-00GVC0 WD-WCAL78165566 232.88 GB (488397168 blocks): OK(unit 5)<br />
Controller: c1<br />
-------------<br />
Driver: 1.50.01.002<br />
Model: 7500-8<br />
FW: FE7X 1.05.00.068<br />
BIOS: BE7X 1.08.00.048<br />
Monitor: ME7X 1.01.00.040<br />
Serial #: F11605A3180167<br />
PCB: Rev3<br />
PCHIP: 1.30-33<br />
ACHIP: 3.20<br />
<br />
<br />
# of units: 2<br />
Unit 0: RAID 5 698.65 GB ( 1465185024 blocks): OK<br />
Unit 4: RAID 5 698.65 GB ( 1465185024 blocks): OK<br />
<br />
# of ports: 8<br />
Port 0: WDC WD2500JB-00GVA0 WD-WMAL71301258 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 1: WDC WD2500JB-00GVA0 WD-WMAL71322705 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 2: WDC WD2500JB-00GVA0 WD-WMAL71945050 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 3: WDC WD2500JB-00GVA0 WD-WMAL71316201 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 4: WDC WD2500JB-00GVC0 WD-WCAL78323749 232.88 GB (488397168 blocks): OK(unit 4)<br />
Port 5: WDC WD3200AAJB-00J3A0 WD-WCAV2V689068 298.09 GB (625142448 blocks): OK(unit 4)<br />
Port 6: WDC WD2500JB-00GVC0 WD-WCAL78234420 232.88 GB (488397168 blocks): OK(unit 4)<br />
Port 7: WDC WD2500JB-00GVC0 WD-WCAL78592213 232.88 GB (488397168 blocks): OK(unit 4)<br />
backup2 /d2#<br />
</pre><br />
<br />
On backup2 look for all ok, no verify.<br />
<br />
<pre><br />
[root@backup3 ~]# sh /root/verify.sh<br />
/c2 Driver Version = 1.26.02.002<br />
/c2 Model = 8006-2LP<br />
/c2 Available Memory = 512KB<br />
/c2 Firmware Version = FE8S 1.05.00.068<br />
/c2 Bios Version = BE7X 1.08.00.048<br />
/c2 Boot Loader Version = ME7X 1.01.00.040<br />
/c2 Serial Number = L018501C6481395<br />
/c2 PCB Version = Rev5<br />
/c2 PCHIP Version = 1.30-66<br />
/c2 ACHIP Version = 3.20<br />
/c2 Total Optimal Units = 1<br />
/c2 Not Optimal Units = 0<br />
<br />
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u1 RAID-1 OK - - - 931.512 ON -<br />
<br />
Port Status Unit Size Blocks Serial<br />
---------------------------------------------------------------<br />
p0 OK u1 931.51 GB 1953525168 WD-WMAW31148820<br />
p1 OK u1 931.51 GB 1953525168 WD-WCATR0277515<br />
<br />
<br />
Ctl Date Severity Alarm Message<br />
------------------------------------------------------------------------------<br />
<br />
Sending start verify message to /c2/u1 ... Done.<br />
<br />
<br />
when done run:<br />
tw_cli /c2 show alarms<br />
<br />
[root@backup3 ~]#<br />
</pre><br />
<br />
Automatically starts the verify, just run <tt>tw_cli /c2 show alarms</tt> as instructed to see the results of the verify.<br />
<br />
<pre><br />
[root@backup1 /data/deprecated]# sh /root/verify.sh<br />
/c0 Driver Version = 2.26.02.010<br />
/c0 Model = 9650SE-8LPML<br />
/c0 Available Memory = 224MB<br />
/c0 Firmware Version = FE9X 4.06.00.004<br />
/c0 Bios Version = BE9X 4.05.00.015<br />
/c0 Boot Loader Version = BL9X 3.08.00.001<br />
/c0 Serial Number = L326025A8270177<br />
/c0 PCB Version = Rev 032<br />
/c0 PCHIP Version = 2.00<br />
/c0 ACHIP Version = 1.90<br />
/c0 Number of Ports = 8<br />
/c0 Number of Drives = 6<br />
/c0 Number of Units = 1<br />
/c0 Total Optimal Units = 1<br />
/c0 Not Optimal Units = 0<br />
/c0 JBOD Export Policy = off<br />
/c0 Disk Spinup Policy = 1<br />
/c0 Spinup Stagger Time Policy (sec) = 1<br />
/c0 Auto-Carving Policy = off<br />
/c0 Auto-Carving Size = 2048 GB<br />
/c0 Auto-Rebuild Policy = on<br />
/c0 Controller Bus Type = PCIe<br />
/c0 Controller Bus Width = 1 lane<br />
/c0 Controller Bus Speed = 2.5 Gbps/lane<br />
<br />
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u0 RAID-5 OK - - 64K 4656.56 ON ON<br />
<br />
Port Status Unit Size Blocks Serial<br />
---------------------------------------------------------------<br />
p0 OK u0 931.51 GB 1953525168 9QJ1Y017<br />
p1 DEVICE-ERROR u0 931.51 GB 1953525168 9QJ1ZN07<br />
p2 OK u0 931.51 GB 1953525168 9QJ2XK1R<br />
p3 OK u0 931.51 GB 1953525168 9QJ2010B<br />
p4 OK u0 1.36 TB 2930277168 6XW0L36T<br />
p5 OK u0 931.51 GB 1953525168 WD-WMATV2444836<br />
p6 NOT-PRESENT - - - -<br />
p7 NOT-PRESENT - - - -<br />
<br />
<br />
Ctl Date Severity Alarm Message<br />
------------------------------------------------------------------------------<br />
c0 [Sat May 12 11:27:15 2012] WARNING Sector repair completed: port=0, LBA=0x6AE571C<br />
c0 [Sat May 12 19:16:21 2012] WARNING Sector repair completed: port=1, LBA=0x40E62A23<br />
c0 [Sat May 12 21:40:56 2012] INFO Verify completed: unit=0<br />
c0 [Mon May 14 00:53:53 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFA7<br />
c0 [Mon May 14 00:58:21 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAA<br />
c0 [Mon May 14 04:35:13 2012] WARNING Sector repair completed: port=0, LBA=0x8FEF2CF<br />
c0 [Mon May 14 04:38:22 2012] WARNING Sector repair completed: port=0, LBA=0x8FEF2D1<br />
c0 [Tue May 15 22:53:46 2012] WARNING Sector repair completed: port=0, LBA=0x13C2622<br />
c0 [Wed May 16 00:39:31 2012] WARNING Sector repair completed: port=0, LBA=0x365A67F<br />
c0 [Wed May 16 00:39:37 2012] WARNING Sector repair completed: port=0, LBA=0x365A685<br />
c0 [Wed May 16 00:47:18 2012] WARNING Sector repair completed: port=0, LBA=0x365A687<br />
c0 [Sat May 19 00:01:44 2012] INFO Verify started: unit=0<br />
c0 [Sat May 19 04:46:20 2012] WARNING Sector repair completed: port=0, LBA=0x365A68E<br />
c0 [Sat May 19 13:37:06 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAC<br />
c0 [Sat May 19 13:37:28 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAE<br />
c0 [Sat May 19 13:37:47 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFB1<br />
c0 [Sat May 19 13:38:00 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFB3<br />
c0 [Sat May 19 21:47:45 2012] INFO Verify completed: unit=0<br />
c0 [Wed May 23 12:21:41 2012] INFO Cache synchronization completed: unit=0<br />
c0 [Fri May 25 00:08:19 2012] WARNING Sector repair completed: port=0, LBA=0x12DA76C<br />
c0 [Fri May 25 00:08:34 2012] WARNING Sector repair completed: port=0, LBA=0x12E4901<br />
c0 [Fri May 25 00:09:33 2012] WARNING Sector repair completed: port=0, LBA=0x12DA773<br />
c0 [Fri May 25 00:39:12 2012] WARNING Sector repair completed: port=0, LBA=0x42C597B<br />
c0 [Sat May 26 00:01:45 2012] INFO Verify started: unit=0<br />
c0 [Sat May 26 00:42:05 2012] WARNING Sector repair completed: port=1, LBA=0x323C1AC<br />
c0 [Sat May 26 00:51:43 2012] WARNING Sector repair completed: port=1, LBA=0x323C1AE<br />
c0 [Sat May 26 01:54:21 2012] WARNING Sector repair completed: port=1, LBA=0x2F0D302<br />
c0 [Sat May 26 02:06:38 2012] WARNING Sector repair completed: port=0, LBA=0x12DA777<br />
c0 [Sat May 26 02:07:21 2012] WARNING Sector repair completed: port=0, LBA=0x12E48FE<br />
c0 [Sat May 26 04:20:00 2012] WARNING Sector repair completed: port=1, LBA=0x2F0D306<br />
c0 [Sat May 26 04:32:58 2012] WARNING Sector repair completed: port=1, LBA=0x323C1B1<br />
c0 [Sat May 26 04:33:21 2012] WARNING Sector repair completed: port=1, LBA=0x323C1B3<br />
c0 [Sat May 26 04:33:44 2012] WARNING Sector repair completed: port=1, LBA=0x323C1BA<br />
c0 [Sat May 26 05:24:07 2012] WARNING Sector repair completed: port=1, LBA=0x3F83862<br />
c0 [Sat May 26 05:25:09 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 06:08:13 2012] WARNING Sector repair completed: port=0, LBA=0x4CDC6A2<br />
c0 [Sat May 26 09:49:35 2012] WARNING Sector repair completed: port=1, LBA=0x6CACD4A<br />
c0 [Sat May 26 18:10:44 2012] WARNING Sector repair completed: port=1, LBA=0x18F425EA<br />
c0 [Sat May 26 19:45:40 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:22:52 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:15 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:22 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:35 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:41 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:49 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:57 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:24:02 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:54:41 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 22:00:30 2012] INFO Verify completed: unit=0<br />
c0 [Sat Jun 2 00:01:43 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 2 00:30:17 2012] WARNING Sector repair completed: port=0, LBA=0x2B911E4<br />
c0 [Sat Jun 2 00:50:57 2012] WARNING Sector repair completed: port=0, LBA=0x5A807CA6<br />
c0 [Sat Jun 2 04:13:13 2012] WARNING Sector repair completed: port=0, LBA=0x2D18291<br />
c0 [Sat Jun 2 04:13:35 2012] WARNING Sector repair completed: port=0, LBA=0x2D1829F<br />
c0 [Sat Jun 2 21:48:02 2012] INFO Verify completed: unit=0<br />
c0 [Mon Jun 4 04:40:34 2012] WARNING Sector repair completed: port=1, LBA=0x4AF8098F<br />
c0 [Tue Jun 5 00:28:19 2012] WARNING Sector repair completed: port=1, LBA=0x263C5CD<br />
c0 [Tue Jun 5 00:33:06 2012] WARNING Sector repair completed: port=1, LBA=0x263C5CF<br />
c0 [Thu Jun 7 00:34:27 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B5F<br />
c0 [Thu Jun 7 00:38:50 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B61<br />
c0 [Fri Jun 8 00:07:13 2012] WARNING Sector repair completed: port=0, LBA=0xC131F6B<br />
c0 [Sat Jun 9 00:01:41 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 9 00:29:11 2012] WARNING Sector repair completed: port=0, LBA=0x6C7614D<br />
c0 [Sat Jun 9 00:38:25 2012] WARNING Sector repair completed: port=0, LBA=0x6C76152<br />
c0 [Sat Jun 9 04:02:30 2012] WARNING Sector repair completed: port=1, LBA=0x263C5D1<br />
c0 [Sat Jun 9 04:02:52 2012] WARNING Sector repair completed: port=1, LBA=0x263C5D3<br />
c0 [Sat Jun 9 04:07:32 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E12<br />
c0 [Sat Jun 9 04:07:57 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E15<br />
c0 [Sat Jun 9 04:08:16 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E17<br />
c0 [Sat Jun 9 04:08:45 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E19<br />
c0 [Sat Jun 9 04:15:04 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B64<br />
c0 [Sat Jun 9 04:15:26 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B66<br />
c0 [Sat Jun 9 04:15:45 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B68<br />
c0 [Sat Jun 9 04:15:59 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B6C<br />
c0 [Sat Jun 9 04:16:13 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B6E<br />
c0 [Sat Jun 9 21:48:52 2012] INFO Verify completed: unit=0<br />
c0 [Thu Jun 14 00:40:10 2012] WARNING Sector repair completed: port=0, LBA=0x334F14B<br />
c0 [Sat Jun 16 00:01:38 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 16 21:16:19 2012] INFO Verify completed: unit=0<br />
c0 [Tue Jun 19 02:03:43 2012] WARNING Sector repair completed: port=1, LBA=0xFE41EAD<br />
c0 [Wed Jun 20 02:30:02 2012] WARNING Sector repair completed: port=1, LBA=0xD99145C<br />
c0 [Sat Jun 23 00:01:36 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 23 04:27:04 2012] WARNING Sector repair completed: port=1, LBA=0x2FAD311<br />
c0 [Sat Jun 23 06:52:38 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC8D<br />
c0 [Sat Jun 23 06:53:03 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC91<br />
c0 [Sat Jun 23 06:53:21 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC94<br />
c0 [Sat Jun 23 17:00:22 2012] WARNING Sector repair completed: port=1, LBA=0xF9AC7C9<br />
c0 [Sat Jun 23 21:15:19 2012] INFO Verify completed: unit=0<br />
c0 [Sat Jun 30 00:01:34 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 30 05:24:13 2012] WARNING Sector repair completed: port=0, LBA=0x3FAA9E7<br />
c0 [Sat Jun 30 14:49:39 2012] WARNING Sector repair completed: port=1, LBA=0x869931C<br />
c0 [Sat Jun 30 21:31:05 2012] INFO Verify completed: unit=0<br />
c0 [Tue Jul 3 03:40:25 2012] WARNING Sector repair completed: port=1, LBA=0xD36C7F7<br />
c0 [Fri Jul 6 02:50:18 2012] WARNING Sector repair completed: port=1, LBA=0x3562470<br />
c0 [Fri Jul 6 22:18:26 2012] WARNING Sector repair completed: port=1, LBA=0x3563173<br />
c0 [Sat Jul 7 00:01:31 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 7 00:50:16 2012] WARNING Sector repair completed: port=0, LBA=0x76EE88<br />
c0 [Sat Jul 7 00:50:39 2012] WARNING Sector repair completed: port=0, LBA=0x76EE8F<br />
c0 [Sat Jul 7 21:39:36 2012] INFO Verify completed: unit=0<br />
c0 [Sun Jul 8 02:51:05 2012] WARNING Sector repair completed: port=0, LBA=0x67759D<br />
c0 [Sun Jul 8 02:53:55 2012] WARNING Sector repair completed: port=0, LBA=0x67759B<br />
c0 [Tue Jul 10 16:17:21 2012] WARNING Sector repair completed: port=0, LBA=0x15C8C695<br />
c0 [Wed Jul 11 22:51:22 2012] WARNING Sector repair completed: port=1, LBA=0x355BBD0<br />
c0 [Sat Jul 14 00:01:28 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 14 01:33:40 2012] WARNING Sector repair completed: port=1, LBA=0x1333BCF4<br />
c0 [Sat Jul 14 03:36:23 2012] WARNING Sector repair completed: port=1, LBA=0x2174773<br />
c0 [Sat Jul 14 11:26:44 2012] WARNING Sector repair completed: port=0, LBA=0x7429AB7<br />
c0 [Sat Jul 14 16:53:50 2012] WARNING Sector repair completed: port=1, LBA=0xA17EB3F<br />
c0 [Sat Jul 14 21:19:25 2012] INFO Verify completed: unit=0<br />
c0 [Wed Jul 18 05:08:47 2012] WARNING Sector repair completed: port=1, LBA=0x17D62EDC<br />
c0 [Wed Jul 18 05:14:15 2012] WARNING Sector repair completed: port=1, LBA=0x17D62EE1<br />
c0 [Thu Jul 19 03:24:59 2012] WARNING Sector repair completed: port=0, LBA=0x7733C3D<br />
c0 [Thu Jul 19 03:25:20 2012] WARNING Sector repair completed: port=0, LBA=0x773CEA5<br />
c0 [Thu Jul 19 03:28:16 2012] WARNING Sector repair completed: port=0, LBA=0x7733C42<br />
c0 [Thu Jul 19 03:28:41 2012] WARNING Sector repair completed: port=0, LBA=0x773CEAF<br />
c0 [Sat Jul 21 00:01:26 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 21 03:07:30 2012] WARNING Sector repair completed: port=1, LBA=0x1CC6936<br />
c0 [Sat Jul 21 03:07:52 2012] WARNING Sector repair completed: port=1, LBA=0x1CC6938<br />
c0 [Sat Jul 21 03:08:11 2012] WARNING Sector repair completed: port=1, LBA=0x1CC693A<br />
c0 [Sat Jul 21 16:43:56 2012] WARNING Sector repair completed: port=0, LBA=0xD04C914<br />
c0 [Sat Jul 21 16:45:31 2012] WARNING Sector repair completed: port=1, LBA=0xD456973<br />
c0 [Sat Jul 21 21:14:29 2012] INFO Verify completed: unit=0<br />
c0 [Wed Jul 25 03:37:25 2012] WARNING Sector repair completed: port=0, LBA=0x1F8E6C43<br />
c0 [Sat Jul 28 00:01:24 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 28 01:45:27 2012] WARNING Sector repair completed: port=0, LBA=0x11584AD<br />
c0 [Sat Jul 28 18:54:25 2012] WARNING Sector repair completed: port=1, LBA=0x447C3E6C<br />
c0 [Sat Jul 28 21:13:46 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 1 03:20:11 2012] WARNING Sector repair completed: port=0, LBA=0x805FEF<br />
c0 [Fri Aug 3 00:50:03 2012] WARNING Sector repair completed: port=0, LBA=0xCED0ACA<br />
c0 [Sat Aug 4 00:01:22 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 4 00:52:51 2012] WARNING Sector repair completed: port=0, LBA=0x805FF3<br />
c0 [Sat Aug 4 00:53:14 2012] WARNING Sector repair completed: port=0, LBA=0x805FF5<br />
c0 [Sat Aug 4 00:53:33 2012] WARNING Sector repair completed: port=0, LBA=0x805FF7<br />
c0 [Sat Aug 4 00:53:47 2012] WARNING Sector repair completed: port=0, LBA=0x805FF9<br />
c0 [Sat Aug 4 00:54:00 2012] WARNING Sector repair completed: port=0, LBA=0x805FFB<br />
c0 [Sat Aug 4 00:54:14 2012] WARNING Sector repair completed: port=0, LBA=0x805FFD<br />
c0 [Sat Aug 4 00:54:27 2012] WARNING Sector repair completed: port=0, LBA=0x805FFF<br />
c0 [Sat Aug 4 04:43:12 2012] WARNING Sector repair completed: port=1, LBA=0x16974289<br />
c0 [Sat Aug 4 04:58:17 2012] WARNING Sector repair completed: port=1, LBA=0x1697428E<br />
c0 [Sat Aug 4 20:54:53 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 8 03:21:55 2012] ERROR Drive timeout detected: port=1<br />
c0 [Wed Aug 8 15:31:44 2012] WARNING Sector repair completed: port=0, LBA=0x1A366CD3<br />
c0 [Sat Aug 11 00:01:21 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 11 20:40:51 2012] INFO Verify completed: unit=0<br />
c0 [Thu Aug 16 05:10:55 2012] WARNING Sector repair completed: port=0, LBA=0x1C22593<br />
c0 [Sat Aug 18 00:01:18 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 18 03:00:20 2012] WARNING Sector repair completed: port=0, LBA=0x1C225A5<br />
c0 [Sat Aug 18 03:43:00 2012] WARNING Sector repair completed: port=1, LBA=0x23EE91E<br />
c0 [Sat Aug 18 03:43:23 2012] WARNING Sector repair completed: port=1, LBA=0x23EE920<br />
c0 [Sat Aug 18 17:00:06 2012] WARNING Sector repair completed: port=1, LBA=0x137D066A<br />
c0 [Sat Aug 18 17:00:29 2012] WARNING Sector repair completed: port=1, LBA=0x137D066D<br />
c0 [Sat Aug 18 21:13:01 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 22 01:36:08 2012] WARNING Sector repair completed: port=0, LBA=0x2560A0F<br />
c0 [Wed Aug 22 01:37:42 2012] WARNING Sector repair completed: port=0, LBA=0x2560A13<br />
c0 [Fri Aug 24 04:01:36 2012] WARNING Sector repair completed: port=1, LBA=0x55C1A5DF<br />
c0 [Fri Aug 24 05:02:06 2012] WARNING Sector repair completed: port=1, LBA=0xCE3378A<br />
c0 [Sat Aug 25 00:01:17 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 25 00:31:06 2012] WARNING Sector repair completed: port=1, LBA=0x50F65D<br />
c0 [Sat Aug 25 00:39:52 2012] WARNING Sector repair completed: port=0, LBA=0x678FF4<br />
c0 [Sat Aug 25 03:43:15 2012] WARNING Sector repair completed: port=0, LBA=0x2560A15<br />
c0 [Sat Aug 25 03:43:39 2012] WARNING Sector repair completed: port=0, LBA=0x2560A19<br />
c0 [Sat Aug 25 03:43:58 2012] WARNING Sector repair completed: port=0, LBA=0x2560A1B<br />
c0 [Sat Aug 25 03:44:30 2012] WARNING Sector repair completed: port=0, LBA=0x2560A21<br />
c0 [Sat Aug 25 20:58:14 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 29 04:57:15 2012] WARNING Sector repair completed: port=1, LBA=0xF3957EB<br />
c0 [Sat Sep 1 00:01:15 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 1 03:21:52 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC86<br />
c0 [Sat Sep 1 03:22:15 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC88<br />
c0 [Sat Sep 1 03:22:34 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC8A<br />
c0 [Sat Sep 1 03:22:47 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC8C<br />
c0 [Sat Sep 1 17:17:22 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD1<br />
c0 [Sat Sep 1 17:17:45 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD3<br />
c0 [Sat Sep 1 17:18:04 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD5<br />
c0 [Sat Sep 1 21:36:56 2012] INFO Verify completed: unit=0<br />
c0 [Thu Sep 6 00:07:30 2012] WARNING Sector repair completed: port=0, LBA=0xDA3C64B<br />
c0 [Thu Sep 6 00:32:56 2012] WARNING Sector repair completed: port=1, LBA=0x6BBA816<br />
c0 [Sat Sep 8 00:01:13 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 8 00:09:56 2012] WARNING Sector repair completed: port=0, LBA=0xDEBC958<br />
c0 [Sat Sep 8 04:38:45 2012] WARNING Sector repair completed: port=0, LBA=0x38D254F<br />
c0 [Sat Sep 8 20:44:50 2012] INFO Verify completed: unit=0<br />
c0 [Mon Sep 10 01:26:34 2012] WARNING Sector repair completed: port=1, LBA=0xFFD8D5E<br />
c0 [Wed Sep 12 00:33:48 2012] WARNING Sector repair completed: port=1, LBA=0xE8DB928<br />
c0 [Wed Sep 12 00:36:33 2012] WARNING Sector repair completed: port=1, LBA=0x6D49411<br />
c0 [Fri Sep 14 01:59:39 2012] WARNING Sector repair completed: port=0, LBA=0x1467F1C<br />
c0 [Fri Sep 14 02:08:27 2012] WARNING Sector repair completed: port=1, LBA=0x14C8ABD<br />
c0 [Fri Sep 14 03:54:47 2012] WARNING Sector repair completed: port=0, LBA=0x1580C915<br />
c0 [Sat Sep 15 00:01:11 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 15 02:38:14 2012] WARNING Sector repair completed: port=0, LBA=0x1C178973<br />
c0 [Sat Sep 15 02:59:02 2012] WARNING Sector repair completed: port=0, LBA=0x1C178975<br />
c0 [Sat Sep 15 04:47:08 2012] WARNING Sector repair completed: port=0, LBA=0x3FA0356<br />
c0 [Sat Sep 15 04:47:31 2012] WARNING Sector repair completed: port=0, LBA=0x3FA0359<br />
c0 [Sat Sep 15 10:41:59 2012] WARNING Sector repair completed: port=0, LBA=0x6DFD1EC<br />
c0 [Sat Sep 15 13:25:23 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD100<br />
c0 [Sat Sep 15 13:25:31 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD104<br />
c0 [Sat Sep 15 13:25:54 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD106<br />
c0 [Sat Sep 15 17:10:50 2012] WARNING Sector repair completed: port=0, LBA=0x1C178977<br />
c0 [Sat Sep 15 20:59:57 2012] INFO Verify completed: unit=0<br />
c0 [Tue Sep 18 01:17:18 2012] WARNING Sector repair completed: port=1, LBA=0x803B05B<br />
c0 [Sat Sep 22 00:01:10 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 22 20:54:31 2012] INFO Verify completed: unit=0<br />
c0 [Tue Sep 25 01:56:47 2012] WARNING Sector repair completed: port=0, LBA=0x26E3909<br />
c0 [Sat Sep 29 00:01:08 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 29 02:04:14 2012] WARNING Sector repair completed: port=0, LBA=0x146AC03<br />
c0 [Sat Sep 29 10:58:39 2012] WARNING Sector repair completed: port=0, LBA=0x6D4EB0E<br />
c0 [Sat Sep 29 10:59:02 2012] WARNING Sector repair completed: port=0, LBA=0x6D4EB14<br />
c0 [Sat Sep 29 11:22:44 2012] WARNING Sector repair completed: port=0, LBA=0x6F79623<br />
c0 [Sat Sep 29 13:50:48 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D65E<br />
c0 [Sat Sep 29 13:51:11 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D661<br />
c0 [Sat Sep 29 13:51:30 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D663<br />
c0 [Sat Sep 29 20:57:34 2012] INFO Verify completed: unit=0<br />
c0 [Mon Oct 1 04:47:24 2012] WARNING Sector repair completed: port=0, LBA=0xC5BC6F2<br />
c0 [Tue Oct 2 02:00:27 2012] WARNING Sector repair completed: port=0, LBA=0x1547667<br />
c0 [Tue Oct 2 02:01:56 2012] WARNING Sector repair completed: port=0, LBA=0x154766F<br />
c0 [Tue Oct 2 05:02:31 2012] WARNING Sector repair completed: port=1, LBA=0xD67D054<br />
c0 [Tue Oct 2 05:04:14 2012] WARNING Sector repair completed: port=1, LBA=0xD67D056<br />
c0 [Wed Oct 3 01:22:12 2012] WARNING Sector repair completed: port=1, LBA=0x12AAF8CA<br />
c0 [Thu Oct 4 04:29:22 2012] WARNING Sector repair completed: port=0, LBA=0x13E6F992<br />
c0 [Thu Oct 4 05:10:51 2012] WARNING Sector repair completed: port=0, LBA=0x1C252A4<br />
c0 [Sat Oct 6 00:01:07 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 6 19:41:18 2012] WARNING Sector repair completed: port=1, LBA=0x5A5C3AE8<br />
c0 [Sat Oct 6 21:01:05 2012] INFO Verify completed: unit=0<br />
c0 [Mon Oct 8 00:32:06 2012] WARNING Sector repair completed: port=0, LBA=0x6C60D3E<br />
c0 [Tue Oct 9 03:51:03 2012] WARNING Sector repair completed: port=1, LBA=0x89B5EC9<br />
c0 [Thu Oct 11 04:21:17 2012] WARNING Sector repair completed: port=1, LBA=0x13F85833<br />
c0 [Sat Oct 13 00:01:05 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 13 05:12:40 2012] WARNING Sector repair completed: port=0, LBA=0x3FA5134<br />
c0 [Sat Oct 13 21:08:35 2012] INFO Verify completed: unit=0<br />
c0 [Tue Oct 16 03:53:50 2012] WARNING Sector repair completed: port=1, LBA=0x148AA1BD<br />
c0 [Thu Oct 18 03:20:30 2012] WARNING Sector repair completed: port=0, LBA=0x1C8DABCB<br />
c0 [Thu Oct 18 04:52:50 2012] WARNING Sector repair completed: port=0, LBA=0xE879057<br />
c0 [Sat Oct 20 00:01:04 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 20 02:19:25 2012] WARNING Sector repair completed: port=1, LBA=0x174B012<br />
c0 [Sat Oct 20 03:41:38 2012] WARNING Sector repair completed: port=0, LBA=0x256D93B<br />
c0 [Sat Oct 20 03:42:01 2012] WARNING Sector repair completed: port=0, LBA=0x256D93D<br />
c0 [Sat Oct 20 03:42:40 2012] WARNING Sector repair completed: port=0, LBA=0x256D940<br />
c0 [Sat Oct 20 03:42:59 2012] WARNING Sector repair completed: port=0, LBA=0x256D942<br />
c0 [Sat Oct 20 03:43:12 2012] WARNING Sector repair completed: port=0, LBA=0x256D944<br />
c0 [Sat Oct 20 03:43:26 2012] WARNING Sector repair completed: port=0, LBA=0x256D948<br />
c0 [Sat Oct 20 16:37:52 2012] WARNING Sector repair completed: port=0, LBA=0xE879060<br />
c0 [Sat Oct 20 16:38:15 2012] WARNING Sector repair completed: port=0, LBA=0xE879062<br />
c0 [Sat Oct 20 21:00:18 2012] INFO Verify completed: unit=0<br />
c0 [Sat Oct 20 23:49:01 2012] WARNING Sector repair completed: port=1, LBA=0x4473E908<br />
c0 [Sun Oct 21 03:42:26 2012] WARNING Sector repair completed: port=1, LBA=0x175BADD5<br />
c0 [Tue Oct 23 01:09:04 2012] WARNING Sector repair completed: port=1, LBA=0x6E524860<br />
c0 [Fri Oct 26 03:21:25 2012] WARNING Sector repair completed: port=0, LBA=0x802C61<br />
c0 [Fri Oct 26 04:22:21 2012] WARNING Sector repair completed: port=0, LBA=0x176353CD<br />
c0 [Sat Oct 27 00:01:03 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 27 00:49:35 2012] WARNING Sector repair completed: port=0, LBA=0x802C65<br />
c0 [Sat Oct 27 17:02:24 2012] WARNING Sector repair completed: port=1, LBA=0xC1FF26D<br />
c0 [Sat Oct 27 17:09:06 2012] WARNING Sector repair completed: port=0, LBA=0xDF621AD<br />
c0 [Sat Oct 27 21:30:57 2012] INFO Verify completed: unit=0<br />
c0 [Tue Oct 30 00:20:46 2012] WARNING Sector repair completed: port=0, LBA=0xE9FE2AB<br />
c0 [Wed Oct 31 02:02:03 2012] WARNING Sector repair completed: port=0, LBA=0x1460C25<br />
c0 [Wed Oct 31 02:04:05 2012] WARNING Sector repair completed: port=0, LBA=0x1460C28<br />
c0 [Thu Nov 1 00:48:34 2012] WARNING Sector repair completed: port=1, LBA=0xA7C92BE<br />
c0 [Thu Nov 1 05:04:45 2012] WARNING Sector repair completed: port=0, LBA=0x1C252C2<br />
<br />
[root@backup1 /data/deprecated]#<br />
</pre><br />
<br />
Look for failed drives and errors. Obviously from the above we need to probably replace drives 0 and 1 and drive 1 is even showing as having problems, yet the RAID array is healthy, amazingly. You also see the automatic verifies.<br />
<br />
Note: when rebuilding a degraded mirror, you will see no progress as it rebuilds in the cli<br />
<pre>Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u0 RAID-5 REBUILDING 0 - 64K 4656.56 OFF ON</pre><br />
<br />
<br />
=== areca ===<br />
<br />
We are using an areca controller on backup3. <br />
<br />
<pre><br />
[root@newbackup3 ~]# sh /root/verify.sh<br />
# Name Raid Name Level Capacity Ch/Id/Lun State<br />
===============================================================================<br />
1 ARC-1160-VOL#00 Raid Set # 00 Raid5 5000.0GB 00/00/00 Checking(19.7%)<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
# Name Disks TotalCap FreeCap DiskChannels State<br />
===============================================================================<br />
1 Raid Set # 00 6 6000.0GB 0.0GB 123456 Checking<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
Date-Time Device Event Type Elapsed Time Errors<br />
===============================================================================<br />
2012-12-05 20:40:58 ARC-1160-VOL#00 Start Checking<br />
2012-12-01 05:06:04 ARC-1160-VOL#00 Complete Init 027:30:45<br />
2012-11-30 01:35:19 ARC-1160-VOL#00 Start Initialize<br />
2026-08-06 01:34:52 H/W MONITOR Raid Powered On<br />
2012-11-30 01:33:36 ARC-1160-VOL#00 Stop Initialization 000:31:48<br />
2012-11-30 01:01:47 ARC-1160-VOL#00 Start Initialize<br />
2026-08-06 00:58:13 H/W MONITOR Raid Powered On<br />
2012-11-30 00:57:26 ARC-1160-VOL#00 Stop Initialization 000:57:07<br />
2012-11-30 00:00:19 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 23:56:48 H/W MONITOR Raid Powered On<br />
2026-08-05 23:52:58 H/W MONITOR Raid Powered On<br />
2026-08-05 23:50:14 H/W MONITOR Raid Powered On<br />
2026-08-05 23:43:30 H/W MONITOR Raid Powered On<br />
2012-11-29 23:10:07 ARC-1160-VOL#00 Stop Initialization 000:00:56<br />
2012-11-29 23:09:11 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 23:08:57 H/W MONITOR Raid Powered On<br />
2012-11-29 23:08:10 ARC-1160-VOL#00 Stop Initialization 000:20:41<br />
2012-11-29 22:47:29 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 22:46:59 H/W MONITOR Raid Powered On<br />
2026-08-05 22:45:55 H/W MONITOR Raid Powered On<br />
2026-08-05 22:44:53 H/W MONITOR Raid Powered On<br />
2026-08-05 22:42:06 H/W MONITOR Raid Powered On<br />
2026-08-05 22:40:50 H/W MONITOR Raid Powered On<br />
2012-11-29 22:40:04 ARC-1160-VOL#00 Stop Initialization 000:24:25<br />
2012-11-29 22:15:38 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 22:15:11 000:000001215B00 Restart Init LBA Point<br />
2026-08-05 22:15:10 H/W MONITOR Raid Powered On<br />
2012-11-29 21:56:38 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 21:56:12 H/W MONITOR Raid Powered On<br />
2026-08-05 21:56:04 IDE Channel #03 Device Inserted<br />
2012-11-29 21:55:13 IDE Channel #04 Device Inserted<br />
2012-11-29 21:55:03 IDE Channel #02 Device Inserted<br />
2026-08-05 21:53:09 H/W MONITOR Raid Powered On<br />
2026-08-05 20:51:46 H/W MONITOR Raid Powered On<br />
2026-08-05 20:49:56 H/W MONITOR Raid Powered On<br />
2026-08-05 20:48:29 H/W MONITOR Raid Powered On<br />
2026-08-05 20:46:29 H/W MONITOR Raid Powered On<br />
2026-08-05 20:44:49 H/W MONITOR Raid Powered On<br />
2026-08-05 20:43:01 H/W MONITOR Raid Powered On<br />
2026-08-05 20:36:25 H/W MONITOR Raid Powered On<br />
2026-08-05 20:31:18 H/W MONITOR Raid Powered On<br />
2026-08-05 20:30:08 H/W MONITOR Raid Powered On<br />
2026-08-05 20:08:40 H/W MONITOR Raid Powered On<br />
2026-08-05 20:06:11 H/W MONITOR Raid Powered On<br />
2026-08-05 20:05:14 H/W MONITOR Raid Powered On<br />
2026-08-05 20:03:58 H/W MONITOR Raid Powered On<br />
2026-08-05 20:00:56 H/W MONITOR Raid Powered On<br />
2026-08-05 19:57:57 H/W MONITOR Raid Powered On<br />
2026-08-05 19:56:15 H/W MONITOR Raid Powered On<br />
2026-08-05 19:55:05 H/W MONITOR Raid Powered On<br />
2026-08-05 17:24:36 H/W MONITOR Raid Powered On<br />
2026-08-05 17:22:43 H/W MONITOR Raid Powered On<br />
2026-08-05 04:50:42 H/W MONITOR Raid Powered On<br />
2026-08-05 04:47:33 H/W MONITOR Raid Powered On<br />
2026-08-05 04:43:57 H/W MONITOR Raid Powered On<br />
2026-08-05 04:18:52 H/W MONITOR Raid Powered On<br />
2026-08-05 04:17:30 H/W MONITOR Raid Powered On<br />
2026-08-05 04:13:30 H/W MONITOR Raid Powered On<br />
2026-08-05 04:10:26 H/W MONITOR Raid Powered On<br />
2026-08-05 04:09:23 H/W MONITOR Raid Powered On<br />
2026-08-05 00:08:09 H/W MONITOR Raid Powered On<br />
2026-08-05 00:07:12 H/W MONITOR Raid Powered On<br />
2026-08-05 00:05:51 H/W MONITOR Raid Powered On<br />
2026-08-05 00:04:27 H/W MONITOR Raid Powered On<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
press enter when ready to run verify<br />
</pre><br />
<br />
Look for failed drives and errors. <br />
<br />
When it proceed's to verifying, you can confirm with:<br />
<br />
<pre><br />
[root@newbackup3 ~]# cli64 vsf info<br />
# Name Raid Name Level Capacity Ch/Id/Lun State<br />
===============================================================================<br />
1 ARC-1160-VOL#00 Raid Set # 00 Raid5 5000.0GB 00/00/00 Checking(22.5%)<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
[root@newbackup3 ~]#<br />
</pre><br />
<br />
== Update OS list ==<br />
<br />
# check for any new VZ templates we want to offer: <tt>vzup2date -z</tt><br />
# see if there's any OS's we want to include in our colo install list. Update 2 places: <tt>signup/html/colo_quote.html</tt> & <tt>signup/html/step1.html</tt><br />
# update the mgmt database (ref_templates table, ref_systems table).<br />
<br />
= Infrequent tasks =<br />
<br />
== Free up space on gateway ==<br />
<br />
<pre>newgateway /var/spool# cd clientmqueue/<br />
newgateway /var/spool/clientmqueue# sh<br />
# for f in `ls`; do rm $f; done<br />
exit</pre><br />
<br />
== Free up space on mail ==<br />
<br />
You can clear out root mail:<br />
<pre><br />
mail /var/log# ll -h /var/mail/root<br />
-rw------- 1 root mail 543K Dec 19 13:05 /var/mail/root<br />
mail /var/log# rm /var/mail/root<br />
</pre><br />
<br />
Or you can archive mail logs:<br />
<pre><br />
mail /var/log# ls -l htt*<br />
-rw-r--r-- 1 root wheel 297436931 Dec 19 13:26 httpd-access.log<br />
-rw-r--r-- 1 root wheel 9824324 Jul 4 11:34 httpd-access.log.old.0.gz<br />
-rw-r--r-- 1 root wheel 6884137 Mar 17 2012 httpd-access.log.old.1.gz<br />
-rw-r--r-- 1 root wheel 18557444 Dec 3 2009 httpd-access.log.old.10.gz<br />
-rw-r--r-- 1 root wheel 14740263 Jan 9 2007 httpd-access.log.old.11.gz<br />
-rw-r--r-- 1 root wheel 14209465 Nov 28 2007 httpd-access.log.old.12.gz<br />
-rw-r--r-- 1 root wheel 16874396 Feb 19 2012 httpd-access.log.old.3.gz<br />
-rw-r--r-- 1 root wheel 14554859 Jul 22 2011 httpd-access.log.old.4.gz<br />
-rw-r--r-- 1 root wheel 10513227 Feb 18 2011 httpd-access.log.old.5.gz<br />
-rw-r--r-- 1 root wheel 7201946 Oct 29 2010 httpd-access.log.old.6.gz<br />
-rw-r--r-- 1 root wheel 10062537 May 6 2010 httpd-access.log.old.7.gz<br />
-rw-r--r-- 1 root wheel 10157042 Aug 12 2010 httpd-access.log.old.8.gz<br />
-rw-r--r-- 1 root wheel 11909534 Mar 4 2010 httpd-access.log.old.9.gz<br />
-rw-r--r-- 1 root wheel 59030930 Dec 19 13:01 httpd-error.log<br />
-rw-r--r-- 1 root wheel 3413134 Mar 4 2010 httpd-error.log.0.gz<br />
-rw-r--r-- 1 root wheel 795515 May 1 2007 httpd-error.log.1.gz<br />
-rw-r--r-- 1 root wheel 1142153 Nov 30 2007 httpd-error.log.2.gz<br />
-rw-r--r-- 1 root wheel 2325801 Feb 18 2011 httpd-error.log.gz<br />
<br />
mail /var/log# sh<br />
# for f in 12 11 10 9 8 7 6 5 4 3 2 1 0; do g=`echo $f+1|bc`; mv httpd-access.log.old.$f.gz httpd-access.log.old.$g.gz; done<br />
# mv httpd-access.log httpd-access.log.old.0<br />
# touch httpd-access.log<br />
# apachectl restart<br />
# gzip httpd-access.log.old.0<br />
<br />
# for f in 2 1 0; do g=`echo $f+1|bc`; mv httpd-error.log.$f.gz httpd-error.log.$g.gz; done<br />
# mv httpd-error.log httpd-error.log.0<br />
# touch httpd-error.log<br />
# apachectl restart<br />
# gzip httpd-error.log.0<br />
# exit<br />
</pre><br />
<br />
== Free up space on bwdb2 ==<br />
<br />
You can either remove items from <tt>/usr/home/archive</tt> or you can scp them to backup3:/data/bwdb2/archive .<br />
<br />
== Free up space on backup1 ==<br />
<br />
backup1 is our primary customer backup system. As usage grows over time, it needs to be regularly purged of old files. The easiest way to do this is by removing deprecated files. These mostly consist of cancelled customers or temporary dump/storage files (created during dump/restores). Our standard policy is to hang onto cancelled customers for 6mos after which we remove their files (as far as customers know their data is purged immediately, but we hang onto it just in case.. and in some cases we cancel a server due to non payment so this makes it easy to restore their system). To find files to remove:<br />
<br />
<pre>[root@backup1 ~]# cd /data/deprecated/<br />
[root@backup1 /data/deprecated]# ls<br />
2101-migrated-20120317.tgz old-683-cxld-20121021.tgz<br />
69.55.230.2-wwwbackup old-744-cxld-20120708.tgz<br />
991-DONTDELETE.tgz old-809-cxld-20120609.tgz<br />
archive-col02050-mdfile-cxld-20120409.gz old-854-cxld-20120621.tgz<br />
col01371.tgz old-931-cxld-20060513.tgz<br />
deleteme_ubuntu-10.10-x86_20111205 old-col00123-mdfile-noarchive-20120417.gz<br />
jail10_old old-col00147-vnfile-cxld-20120828.gz<br />
jail14_rsync_old old-col00419-dump-cxld-20120224.gz<br />
jail15_old old-col01098-vnfile-cxld-20120827.gz<br />
jail3_old old-col01278-dump-cxld-20120822<br />
jail4_old old-col01517-dump-cxld-20120828<br />
jail5_old old-col01669-dump-cxld-20120203.gz<br />
old-1009-cxld-20120608.tgz old-col01687-dump-cxld-20120909<br />
old-1012-cxld-20120411.tgz old-col01790-dump-cxld-20120828<br />
old-1052-cxld-20120721.tgz old-col01812-dump-cxld-20120820<br />
old-10631-cxld-20120622.tgz old-col01938-mdfile-cxld-20120619.gz<br />
old-10632-cxld-20120622.tgz old-col02095-mdfile-noarchive-20120523.gz<br />
old-10633-cxld-20120622.tgz olddebian-3.0-v15-20110610.tgz<br />
old-1236-cxld-20120621.tgz oldmod_frontpage-deb30-v15-20110610.tgz<br />
old-1381-cxld-20120404.tgz oldmod_perl-deb30-v15-20110610.tgz<br />
old-1422-cxld-20120721.tgz oldmod_ssl-deb30-v15-20110610.tgz<br />
old-14681-cxld-20120619.tgz oldmysql-deb30-v15-20110610.tgz<br />
old-1544-cxld-20120626.tgz oldproftpd-deb30-v15-20110610.tgz<br />
old-18351-cxld-20120605.tgz old_virt14<br />
old-1853-cxld-20120910.tgz old_virt18<br />
old-1963-cxld-20120206.tgz oldwebmin-deb30-v15-20110610.tgz<br />
old-1967-cxld-20120605.tgz suse.virt11.20120421.tgz<br />
old-1981-noarchive-20120729.tgz virt11<br />
old-2030-migrated-noarchive-20120727.tgz virt12_old<br />
old-2037-cxld-20120716.tgz virt13_old<br />
old-2065-cxld-20120727.tgz virt16_old<br />
old-2068-cxld-20120424.tgz virt4_old<br />
old-2085-cxld-20120531.tgz virt5_old<br />
old-364-cxld-20120904.tgz virt6_old<br />
old-446-cxld-20120512.tgz virt7_old<br />
old-613-cxld-20120601.tgz virt8_old<br />
[root@backup1 /data/deprecated]#<br />
</pre><br />
<br />
virtX_old and jailX_old are permanently archived, so ignore those as well as anything else marked not to delete or otherwise suspicious. Likewise, probably a good idea to try to hang onto oldTEMPLATE.gz as long as we can as well.<br />
Most of the stuff we want to delete is dated when it was deprecated, making this easy. So to remove files from 6 mos ago (running this in Oct):<br />
<br />
<pre>[root@backup1 /data/deprecated]# ls old*201204*<br />
old-1012-cxld-20120411.tgz old-2068-cxld-20120424.tgz<br />
old-1381-cxld-20120404.tgz old-col00123-mdfile-noarchive-20120417.gz<br />
[root@backup1 /data/deprecated]# rm old*201204*<br />
</pre><br />
<br />
Every few months you will also want to remove some of the snapshot archives for mail. We typically save the 1st, 10th, and 20th of each month. To do this you set aside the dates you want to save then remove months at a time, followed by restoring the set aside dates. Here's how that works:<br />
<pre>[root@backup1 /data/www/daily]# ls<br />
05 08-10-11 10-04-10 11-10-10 12-07-29 12-09-21 12-11-14<br />
06 08-10-21 10-04-20 11-10-20 12-07-30 12-09-22 12-11-15<br />
06-06-01-usr-home.tgz 08-11-01 10-05-01 11-11-01 12-07-31 12-09-23 12-11-16<br />
06-07-01-usr-home.tgz 08-11-10 10-05-11 11-11-10 12-08-01 12-09-24 12-11-17<br />
06-08-01-usr-home.tgz 08-11-20 10-05-20 11-11-20 12-08-02 12-09-25 12-11-18<br />
06-09-01-usr-home.tgz 08-12-01 10-06-01 11-12-01 12-08-03 12-09-26 12-11-19<br />
06-11-10 08-12-10 10-06-10 11-12-10 12-08-04 12-09-27 12-11-20<br />
06-12-21 08-12-20 10-06-20 11-12-20 12-08-05 12-09-28 12-11-21<br />
07-01-10 09-01-01 10-07-01 12-01-01 12-08-06 12-09-29 12-11-22<br />
07-01-20 09-01-10 10-07-10 12-01-10 12-08-07 12-09-30 12-11-23<br />
07-02-10 09-01-20 10-07-20 12-01-20 12-08-08 12-10-01 12-11-24<br />
07-02-20 09-02-01 10-08-01 12-02-01 12-08-09 12-10-02 12-11-25<br />
07-03-01 09-02-10 10-08-10 12-02-10 12-08-10 12-10-03 12-11-26<br />
07-03-20 09-02-20 10-08-20 12-02-20 12-08-11 12-10-04 12-11-27<br />
07-04-01 09-03-01 10-09-01 12-03-01 12-08-12 12-10-05 12-11-28<br />
07-04-10 09-03-10 10-09-10 12-03-10 12-08-13 12-10-06 12-11-29<br />
07-04-20 09-03-20 10-09-20 12-03-20 12-08-14 12-10-07 12-11-30<br />
07-05-01 09-04-01 10-10-01 12-04-01 12-08-15 12-10-08 12-12-01<br />
07-05-10 09-04-10 10-10-10 12-04-10 12-08-16 12-10-09 12-12-02<br />
07-05-20 09-04-20 10-10-20 12-04-20 12-08-17 12-10-10 12-12-03<br />
07-06-01 09-05-01 10-11-01 12-05-01 12-08-18 12-10-11 12-12-04<br />
07-06-10 09-05-10 10-11-10 12-05-10 12-08-19 12-10-12 12-12-05<br />
07-06-20 09-05-20 10-11-20 12-05-20 12-08-20 12-10-13 12-12-06<br />
07-07-20 09-06-01 10-12-01 12-06-01 12-08-21 12-10-14 12-12-07<br />
07-08-10 09-06-10 10-12-10 12-06-10 12-08-22 12-10-15 12-12-08<br />
07-08-20 09-06-20 10-12-20 12-06-20 12-08-23 12-10-16 12-12-09<br />
07-09-01 09-07-01 11-01-01 12-07-01 12-08-24 12-10-17 12-12-10<br />
07-10-01 09-07-10 11-01-10 12-07-02 12-08-25 12-10-18 12-12-11<br />
07-10-10 09-07-20 11-01-21 12-07-03 12-08-26 12-10-19 12-12-12<br />
07-10-20 09-08-01 11-02-01 12-07-04 12-08-27 12-10-20 12-12-13<br />
07-12-01 09-08-10 11-02-10 12-07-05 12-08-28 12-10-21 12-12-14<br />
07-12-10 09-08-20 11-02-20 12-07-06 12-08-29 12-10-22 12-12-15<br />
08-01-01 09-09-01 11-03-01 12-07-07 12-08-30 12-10-23 12-12-16<br />
08-01-20 09-09-10 11-03-10 12-07-08 12-08-31 12-10-24 12-12-17<br />
08-02-20 09-09-20 11-03-20 12-07-09 12-09-01 12-10-25 12-12-18<br />
08-03-01 09-10-01 11-04-01 12-07-10 12-09-02 12-10-26 12-12-19<br />
08-03-10 09-10-10 11-04-10 12-07-11 12-09-03 12-10-27 12-12-20<br />
08-03-20 09-10-20 11-04-20 12-07-12 12-09-04 12-10-28 12-12-21<br />
08-04-01 09-11-01 11-05-01 12-07-13 12-09-05 12-10-29 12-12-22<br />
08-04-20 09-11-10 11-05-10 12-07-14 12-09-06 12-10-30 12-12-23<br />
08-05-01 09-11-20 11-05-20 12-07-15 12-09-07 12-10-31 12-12-24<br />
08-05-10 09-12-01 11-06-01 12-07-16 12-09-08 12-11-01 12-12-25<br />
08-06-10 09-12-10 11-06-10 12-07-17 12-09-09 12-11-02 12-12-26<br />
08-06-20 09-12-20 11-06-20 12-07-18 12-09-10 12-11-03 12-12-27<br />
08-07-02 10-01-01 11-07-01 12-07-19 12-09-11 12-11-04 12-12-28<br />
08-07-10 10-01-10 11-07-10 12-07-20 12-09-12 12-11-05 2008-10-23<br />
08-07-20 10-01-20 11-07-20 12-07-21 12-09-13 12-11-06 bb.tgz<br />
08-08-01 10-02-01 11-08-01 12-07-22 12-09-14 12-11-07 boot<br />
08-08-10 10-02-10 11-08-10 12-07-23 12-09-15 12-11-08 current<br />
08-08-21 10-02-20 11-08-20 12-07-24 12-09-16 12-11-09 hold<br />
08-09-01 10-03-01 11-09-01 12-07-25 12-09-17 12-11-10<br />
08-09-10 10-03-10 11-09-10 12-07-26 12-09-18 12-11-11<br />
08-09-21 10-03-20 11-09-20 12-07-27 12-09-19 12-11-12<br />
08-10-01 10-04-01 11-10-01 12-07-28 12-09-20 12-11-13<br />
[root@backup1 /data/www/daily]#<br />
</pre><br />
<br />
So we see that everything up to July 2012 has been pruned. To prune July 2012 we do the following:<br />
<br />
<pre>mv 12-07-01 hold<br />
mv 12-07-10 hold<br />
mv 12-07-20 hold<br />
rm -fr 12-07*<br />
mv hold/* .</pre></div>75.80.128.147https://wiki.jcihosting.com/index.php?title=Routine_Maintenance&diff=2206Routine Maintenance2020-02-25T18:36:11Z<p>75.80.128.147: /* Outgoing */</p>
<hr />
<div>= Daily Tasks =<br />
<br />
== check load graphs == <br />
<br />
Click on the Load link in mgmt <br />
<br />
This screen shows you load levels on our servers and network traffic for critical machines (firewalls, backup servers).<br />
<br />
If you see load high or increasing <br />
<br />
FreeBSD: <br />
run [[VPS_Management#jtop|jtop]] (or [[VPS_Management#jt|jt]] > 7.x) and see if there are any runaway processes. Here are some examples of entries in top that are<br />
definitely runaway processes:<br />
<br />
<pre>79481 root 64 0 2256K 1056K CPU1 1 58:16 87.40% 87.40% nano<br />
50650 1000 64 0 1852K 1112K RUN 0 207.9H 84.08% 84.08% screen<br />
14829 www 2 0 39100K 31736K accept 0 104:24 46.54% 6.54% httpd<br />
42065 root 61 0 1300K 844K RUN 1 47.8H 91.36% 91.36% ee<br />
1328 www 56 0 18440K 10796K CPU1 0 64.4H 97.71% 97.71% httpd<br />
26251 user 57 0 6124K 1160K CPU1 1 82.9H 98.44% 98.44% screen<br />
89874 root 60 0 1352K 892K RUN 1 33.8H 65.82% 65.82% dialog<br />
38656 1000 64 0 3088K 2136K CPU0 0 806:13 97.95% 97.95% StutBot<br />
27630 root 64 0 1396K 972K RUN 1 76.8H 86.47% 86.47% ee</pre><br />
<br />
Linux:<br />
run [[VPS_Management#vwe|vwe]] to see which VPS’s have high loads. From there run <tt>[[VPS_Management#vp|vp]] <veid></tt> and/or <tt>[[VPS_Management#vt|vt]] <veid></tt> to see what's going on in that system.<br />
[[VPS_Management#vzstat|vzstat]] will also give you a nice picture of whats going on, systems with high numbers in the mlat column are likely culprits.<br />
<br />
examples of out of control procs:<br />
<br />
<pre>12183 nobody 16 0 4916 1348 1340 R 45.5 0.0 4249m httpd<br />
29266 #502 16 0 1852 796 792 R 22.5 0.0 1104m vim<br />
23860 #41 16 0 5472 5472 2076 R 98.9 0.2 31:41 python<br />
19227 bin 19 0 1688 716 652 R 99.9 0.0 321:08 wtrs_ui<br />
7762 apache 16 0 268 236 224 R 85.7 0.0 1010m ptrace<br />
4624 #501 20 0 4304 2400 2044 R 53.6 0.1 284:32 YoSucker<br />
20451 #506 20 0 1876 820 816 R 17.2 0.0 169:35 vim<br />
8834 #514 20 0 900 724 672 R 77.6 0.0 382:30 neostats<br />
31815 apache 14 0 3176 3176 1696 R 74.4 0.1 6:15 counter</pre><br />
<br />
Just kill -9 them and be done with it.<br />
<br />
Also, anytime you see `kmod` or `ptrace` - kill those immediaely no matter how much they are using - they are attempts to exploit the linux ptrace bug. They won't work, but they suck a lot of CPU...<br />
<br />
Also, any other processes that are at 90-100% cpu usage and have been running for any long period of time should be killed except for mysqld processes on FreeBSD. See above.<br />
<br />
However, there is an exception:<br />
<br />
if it is a mysqld, we don't want to kill their database. What you want to do is <tt>[[VPS_Management#jpid|jpid]] <pid></tt> to see who owns it, and then email them the paste containing the instructions for the nanny. Or you can simply do a <tt>kill -1 PID</tt> on the process to restart it.<br />
<br />
=== Load averages jump at night ===<br />
<br />
The load averages on the FreeBSD systems may jump up at night between 1 and 4 am - this is because the backups are running - if this is what is causing a jump in load, you will see processes like `rsync` in top eating a lot of CPU time.<br />
<br />
== check backups ==<br />
<br />
mgmt -> Motnroing -> Backups and make sure every machine was backed up the previous nite. <br />
Also look at df on backup1 and backup2 to make sure no disk is approaching full, though bb should warn us in advance. Please note - errors encountered when a backup script on any of the particular systems run will gnerate an email to support@johncompanies.com so you can know immediately the day after if the directory to be backed up has been moved or no longer exists. A paste exists for this to notify the customer of a non-existant file/dir.<br />
<br />
== check bb for warnings ==<br />
mgmt -> BigBrother<br />
<br />
Some events don't generate pages (on purpose). You will only see them by going to the bb main page.<br />
<br />
== check jail5 for crashed VPSs ==<br />
<br />
On jail5<br />
<br />
notrunning<br />
<br />
To restart a VPS <br />
<br />
vm restart col0XXXX<br />
<br />
== Check NetHere ==<br />
<br />
Check the NetHere servers<br />
<br />
=== Mail systems ===<br />
Check for possible SPAMMERs.<br />
<br />
==== Incoming ====<br />
Check mta-1 and mta-2 count of customer logins for possible<br />
customer SPAM compromises.<br />
<br />
login_count /logs/maillog | tail -30<br />
<br />
==== Outgoing ====<br />
<br />
Check outgoing queues on relay-1 and relay-2<br />
<br />
mail_count | tail -30<br />
<br />
To clean up outgoing queues of unwanted SPAM on relay-1 and relay-2.<br />
<br />
mail_cleanup [ <sender's domain/username/message id> ... ]<br />
<br />
To just remove emails from some senders.<br />
<br />
rmmails <sender's domain/username/message id> ...<br />
<br />
=== Nagios ===<br />
<br />
Check for other problems on NetHere servers<br />
<br />
https://nagios.nethere.net<br />
<br />
=== Cacti ===<br />
<br />
Check bandwidth usage on servers<br />
<br />
https://cacti.nethere.net<br />
<br />
= Monthly Tasks =<br />
== rotate pine sent mail (1st of month) ==<br />
On the 1st of the month, before any emails are sent out, quit out of pine, then log back in. Send mail from last month will be archived.<br />
If you mess up and do it on the 3rd (for example), you can go into the previous month's saved email and save emails from the current month into the <tt>sent-mail</tt> (current month) mailbox.<br />
<br />
== b/w caps ==<br />
On the 1st: remove any bwcaps put into the firewall (only really applies if a bwcap was added cause someone went over on b/w):<br />
<br />
ipfw list|grep pipe<br />
ipfw del [each rule listed]<br />
<br />
NOTE: this cronjob on newgateway will do some of that for you, provided you used one of the following pipe #s:<br />
0 0 1 * * /sbin/ipfw del 3 4 5 17331<br />
<br />
-------------------<br />
<br />
We really don’t do this anymore since we have centralized traffic accounting with netflow, but for posterity:<br />
<br />
Make sure all machines reset counters to 0 after midnight on the 1st<br />
Make sure they dumped a counter<br />
<br />
On each jail run:<br />
trafficgather.pl<br />
<br />
And on each virt:<br />
linuxtrafficgather.pl<br />
<br />
== Monthly RAID checks ==<br />
<br />
Every month we check the health of and verfy the parity on all our RAID-based systems.<br />
To facilitate this, we've created a simple script to start the process:<br />
<br />
sh /root/verify.sh<br />
<br />
=== Adaptec controllers ===<br />
Here's some sample output:<br />
<pre><br />
mail /usr/local/www/scripts# sh /root/verify.sh<br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> container list /f<br />
Executing: container list /full=TRUE<br />
Num Total Oth Chunk Scsi Partition<br />
Creation System<br />
Label Type Size Ctr Size Usage B:ID:L Offset:Size State RO Lk Task Done% Ent<br />
Date Time Files<br />
----- ------ ------ --- ------ ------- ------ ------------- ------- -- -- ------- ------ ---<br />
------ -------- ------<br />
0 Mirror 33.9GB Open 0:01:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:32<br />
/dev/aacd0 mirror0 0:00:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:32<br />
<br />
1 Mirror 33.9GB Open 0:02:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:50<br />
/dev/aacd1 mirror1 0:03:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:50<br />
<br />
<br />
AAC0> disk list /f<br />
Executing: disk list /full=TRUE<br />
<br />
B:ID:L Device Type Removable media Vendor-ID Product-ID Rev Blocks Bytes/Bl<br />
ock Usage Shared Rate<br />
------ -------------- --------------- --------- ---------------- ----- --------- --------<br />
--- ---------------- ------ ----<br />
0:00:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:01:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:02:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:03:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
<br />
AAC0> disk show smart<br />
Executing: disk show smart<br />
<br />
Smart Method of Enable<br />
Capable Informational Exception Performance Error<br />
B:ID:L Device Exceptions(MRIE) Control Enabled Count<br />
------ ------- ---------------- --------- ----------- ------<br />
0:00:0 Y 6 Y N 0<br />
0:01:0 Y 6 Y N 0<br />
0:02:0 Y 6 Y N 0<br />
0:03:0 Y 6 Y N 0<br />
0:06:0 N<br />
<br />
AAC0> task list<br />
Executing: task list<br />
<br />
Controller Tasks<br />
<br />
TaskId Function Done% Container State Specific1 Specific2<br />
------ -------- ------- --------- ----- --------- ---------<br />
<br />
No tasks currently running on controller<br />
<br />
AAC0> dia sh hi<br />
Executing: diagnostic show history<br />
No switches specified, defaulting to "/current".<br />
<br />
<br />
<br />
*** HISTORY BUFFER FROM CURRENT CONTROLLER RUN ***<br />
<br />
[00]: GetDiskLogEntry: container - 1, entry return 0<br />
[01]: Container 1 started SCRUB task<br />
[02]: Starting Mirror:1 scrub<br />
[03]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[04]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[05]: UpdateDiskLogIndex - Set - container 0, index 1<br />
[06]: GetDiskLogEntry: container - 0, entry return 1<br />
[07]: Container 0 started SCRUB task<br />
[08]: Starting Mirror:0 scrub<br />
[09]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[10]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[11]: Mirror Scrub Container:1 ErrorsFound:0<br />
[12]: Clear disk log: sector - 80, driveno 2<br />
[13]: Clear disk log: sector - 80, driveno 3<br />
[14]: Container 1 completed SCRUB task:<br />
[15]: Mirror Scrub Container:0 ErrorsFound:0<br />
[16]: Clear disk log: sector - 81, driveno 1<br />
[17]: Clear disk log: sector - 81, driveno 0<br />
[18]: Container 0 completed SCRUB task:<br />
[19]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[20]: GetDiskLogEntry: container - 0, entry return 0<br />
[21]: Container 0 started SCRUB task<br />
[22]: Starting Mirror:0 scrub<br />
[23]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[24]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[25]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[26]: GetDiskLogEntry: container - 1, entry return 1<br />
[27]: Container 1 started SCRUB task<br />
[28]: Starting Mirror:1 scrub<br />
[29]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[30]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[31]: Mirror Scrub Container:1 ErrorsFound:0<br />
[32]: Clear disk log: sector - 81, driveno 2<br />
[33]: Clear disk log: sector - 81, driveno 3<br />
[34]: Container 1 completed SCRUB task:<br />
[35]: Mirror Scrub Container:0 ErrorsFound:0<br />
[36]: Clear disk log: sector - 80, driveno 1<br />
[37]: Clear disk log: sector - 80, driveno 0<br />
[38]: Container 0 completed SCRUB task:<br />
[39]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[40]: GetDiskLogEntry: container - 0, entry return 0<br />
[41]: Container 0 started SCRUB task<br />
[42]: Starting Mirror:0 scrub<br />
[43]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[44]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[45]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[46]: GetDiskLogEntry: container - 1, entry return 1<br />
[47]: Container 1 started SCRUB task<br />
[48]: Starting Mirror:1 scrub<br />
[49]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[50]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[51]: Mirror Scrub Container:1 ErrorsFound:0<br />
[52]: Clear disk log: sector - 81, driveno 2<br />
[53]: Clear disk log: sector - 81, driveno 3<br />
[54]: Container 1 completed SCRUB task:<br />
[55]: Mirror Scrub Container:0 ErrorsFound:0<br />
[56]: Clear disk log: sector - 80, driveno 1<br />
[57]: Clear disk log: sector - 80, driveno 0<br />
[58]: Container 0 completed SCRUB task:<br />
[59]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[60]: GetDiskLogEntry: container - 0, entry return 0<br />
[61]: Container 0 started SCRUB task<br />
[62]: Starting Mirror:0 scrub<br />
[63]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[64]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[65]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[66]: GetDiskLogEntry: container - 1, entry return 1<br />
[67]: Container 1 started SCRUB task<br />
[68]: Starting Mirror:1 scrub<br />
[69]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[70]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[71]: Mirror Scrub Container:1 ErrorsFound:0<br />
[72]: Clear disk log: sector - 81, driveno 2<br />
[73]: Clear disk log: sector - 81, driveno 3<br />
[74]: Container 1 completed SCRUB task:<br />
[75]: Mirror Scrub Container:0 ErrorsFound:0<br />
[76]: Clear disk log: sector - 80, driveno 1<br />
[77]: Clear disk log: sector - 80, driveno 0<br />
[78]: Container 0 completed SCRUB task:<br />
[79]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[80]: GetDiskLogEntry: container - 0, entry return 0<br />
[81]: Container 0 started SCRUB task<br />
[82]: Starting Mirror:0 scrub<br />
[83]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[84]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[85]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[86]: GetDiskLogEntry: container - 1, entry return 1<br />
[87]: Container 1 started SCRUB task<br />
[88]: Starting Mirror:1 scrub<br />
[89]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[90]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[91]: Mirror Scrub Container:1 ErrorsFound:0<br />
[92]: Clear disk log: sector - 81, driveno 2<br />
[93]: Clear disk log: sector - 81, driveno 3<br />
[94]: Container 1 completed SCRUB task:<br />
[95]: Mirror Scrub Container:0 ErrorsFound:0<br />
[96]: Clear disk log: sector - 80, driveno 1<br />
[97]: Clear disk log: sector - 80, driveno 0<br />
[98]: Container 0 completed SCRUB task:<br />
[99]:<br />
<br />
========================<br />
History Output Complete.<br />
<br />
AAC0><br />
AAC0> exit<br />
Executing: exit<br />
<br />
press enter when ready to run verify <INS><br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> contai scr 0<br />
Executing: container scrub 0<br />
<br />
AAC0> contai scr 1<br />
Executing: container scrub 1<br />
<br />
AAC0> exit<br />
Executing: exit<br />
<br />
when done run: <br />
<br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
<br />
<br />
Nov 1 10:32:46 mail /kernel: aac0: **Monitor** Container 0 started SCRUB task<br />
Nov 1 10:32:47 mail /kernel: aac0: **Monitor** Container 1 started SCRUB task<br />
<br />
</pre><br />
<br />
Here's an analysis of what we're seeing and what we're looking for:<br />
<br />
<pre><br />
AAC0> container list /f<br />
Executing: container list /full=TRUE<br />
Num Total Oth Chunk Scsi Partition<br />
Creation System<br />
Label Type Size Ctr Size Usage B:ID:L Offset:Size State RO Lk Task Done% Ent<br />
Date Time Files<br />
----- ------ ------ --- ------ ------- ------ ------------- ------- -- -- ------- ------ ---<br />
------ -------- ------<br />
0 Mirror 33.9GB Open 0:01:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:32<br />
/dev/aacd0 mirror0 0:00:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:32<br />
<br />
1 Mirror 33.9GB Open 0:02:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:50<br />
/dev/aacd1 mirror1 0:03:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:50<br />
</pre><br />
<br />
This is showing you the health of the arrays. You're looking for ''Normal'' under the State column, and the absence of a ! in the sector size - sometimes, you'll see this:<br />
64.0KB!33.9GB <br />
<br />
That indicates a problem.<br />
<br />
<pre><br />
AAC0> disk show smart<br />
Executing: disk show smart<br />
<br />
Smart Method of Enable<br />
Capable Informational Exception Performance Error<br />
B:ID:L Device Exceptions(MRIE) Control Enabled Count<br />
------ ------- ---------------- --------- ----------- ------<br />
0:00:0 Y 6 Y N 0<br />
0:01:0 Y 6 Y N 0<br />
0:02:0 Y 6 Y N 0<br />
0:03:0 Y 6 Y N 0<br />
0:06:0 N<br />
</pre><br />
<br />
This shows you a SMART report output. Looking for values in the Error Count column.<br />
<pre><br />
AAC0> task list<br />
Executing: task list<br />
<br />
Controller Tasks<br />
<br />
TaskId Function Done% Container State Specific1 Specific2<br />
------ -------- ------- --------- ----- --------- ---------<br />
<br />
No tasks currently running on controller<br />
</pre><br />
<br />
Look for absence of tasks running- a bad thing would be to see a rebuild or verify running when you didn't initiate it.<br />
<br />
With the history output, you're looking for any anomalies or events since the last time a verify was run. If you see a drive with lots of problems, you may want to take backups before allowing the verify to run since it could replicate errors onto the good drive.<br />
<br />
After you see the history output, it will prompt you to press enter to run the verify. If you're happy with all the output you're seeing- mirror is healthy, history looks good, it's safe to proceed. Otherwise ^C to exit. After hitting enter it will start the verify and start to tail the messages log file (so you can easily see when the verify is complete). Here's what that'll look like:<br />
<br />
<pre><br />
Nov 1 14:38:08 mail /kernel: aac0: **Monitor** Container 1 completed SCRUB task:<br />
Nov 1 14:46:45 mail /kernel: aac0: **Monitor** Container 0 completed SCRUB task:<br />
</pre><br />
<br />
So, putting it all together, after hitting enter to start the verify, you'll see:<br />
<br />
<pre><br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> contai scr 0<br />
Executing: container scrub 0<br />
<br />
AAC0> contai scr 1<br />
Executing: container scrub 1<br />
<br />
AAC0> exit<br />
Executing: exit<br />
<br />
when done run: <br />
<br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
<br />
<br />
Nov 1 10:32:46 mail /kernel: aac0: **Monitor** Container 0 started SCRUB task<br />
Nov 1 10:32:47 mail /kernel: aac0: **Monitor** Container 1 started SCRUB task<br />
<br />
</pre><br />
<br />
When the scrub(s) (verify) are complete - if the server has multiple logical drives, it will run both in parallel - you should exit the tail of the log file (^C) and run:<br />
<br />
<pre><br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
</pre><br />
<br />
Which will show you the diagnostic history, you're looking for the results of the most recent scrub:<br />
<br />
<pre><br />
[100]: Mirror Scrub Container:1 ErrorsFound:0<br />
[101]: Clear disk log: sector - 81, driveno 2<br />
[102]: Clear disk log: sector - 81, driveno 3<br />
[103]: Container 1 completed SCRUB task:<br />
[104]: Mirror Scrub Container:0 ErrorsFound:0<br />
[105]: Clear disk log: sector - 80, driveno 1<br />
[106]: Clear disk log: sector - 80, driveno 0<br />
[107]: Container 0 completed SCRUB task:<br />
</pre><br />
<br />
^C to exit the RAID CLI.<br />
<br />
If you see:<br />
<br />
[104]: Mirror Scrub Container:0 ErrorsFound:5<br />
<br />
You'll want to rerun the verify on that drive till it shows 0, or perhaps replace the drive- you should be able to see from the output which drive had the problem.<br />
<br />
Depending on the size and how busy the drive is, the verify can take anywhere from an hour to the better part of a day.<br />
<br />
You will notice that the diagnostic history is not shown on our modern adaptec cards (i.e. any adaptec card not in a Dell 2450). The reason for this is the history is never cleared, so there's simply too much data to show and it just crashes the CLI. So, don't bother trying to see it...which does make it hard to see if there are problems going on, so you just need to watch the scrub to see it goes to 100%. You will also notice that on some servers there's no tail of messages. Again, this is cause no data is shown there about the completion of the scrub. The thing to do here is to go into the CLI and continue to <tt>show tasks</tt> to monitor scrub progress.<br />
<br />
See [[RAIC_CLI#Adaptec|Adaptec RAID CLI Reference]] for more details on how to use the CLI<br />
<br />
=== DELL (LSI-based) SAS controllers ===<br />
<br />
Here's what the output looks like when running verify.sh on a LSI-based card:<br />
<br />
<pre><br />
jail2 /mnt/data2# sh /root/verify.sh<br />
<br />
Adapter #0<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 0<br />
Device Id: 0<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018396142<br />
SAS Address(1): 0x0<br />
Connected Port Number: 0(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DESN<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 1<br />
Device Id: 1<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018395db2<br />
SAS Address(1): 0x0<br />
Connected Port Number: 1(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DERV<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 2<br />
Device Id: 2<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50006eece89<br />
SAS Address(1): 0x0<br />
Connected Port Number: 2(path0)<br />
Inquiry Data: SEAGATE ST3300555SS T2113LM4BFBZ<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 3<br />
Device Id: 3<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50006eee035<br />
SAS Address(1): 0x0<br />
Connected Port Number: 3(path0)<br />
Inquiry Data: SEAGATE ST3300555SS T2113LM4BGF7<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 4<br />
Device Id: 4<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50004bd7ea5<br />
SAS Address(1): 0x0<br />
Connected Port Number: 4(path0)<br />
Inquiry Data: SEAGATE ST3300656SS HS093QP0G8SW<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 5<br />
Device Id: 5<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e01f1c4112<br />
SAS Address(1): 0x0<br />
Connected Port Number: 5(path0)<br />
Inquiry Data: FUJITSU MBA3300RC D306BJ15P9201W06<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 0 (Target Id: 0)<br />
Name:<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:139392MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 1 (Target Id: 1)<br />
Name:MIRROR1<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 2 (Target Id: 2)<br />
Name:MIRROR2<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
Battery FRU : N/A<br />
Battery Warning : Enabled<br />
Memory Correctable Errors : 0<br />
Memory Uncorrectable Errors : 0<br />
BBU : Present<br />
BBU : Yes<br />
Cache When BBU Bad : Disabled<br />
press enter when ready to run verify<br />
<br />
</pre><br />
<br />
Before pressing enter, here's what we're looking for:<br />
<br />
<pre><br />
Enclosure Device ID: 32<br />
Slot Number: 0<br />
Device Id: 0<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018396142<br />
SAS Address(1): 0x0<br />
Connected Port Number: 0(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DESN<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
</pre><br />
<br />
This is the output shown for each physical drive in the system. We're looking to confirm it's Firmware state is ''Online'', and Media Error Count, Other Error Count, and Predictive Failure Count are all zero (or near zero).<br />
<br />
<pre><br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 1 (Target Id: 1)<br />
Name:MIRROR1<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
</pre><br />
<br />
This is the output for each logical drive. We're looking for State ''Optimal''. Also confirm <tt>Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU</tt><br />
<br />
<pre><br />
Exit Code: 0x00<br />
Battery FRU : N/A<br />
Battery Warning : Enabled<br />
Memory Correctable Errors : 0<br />
Memory Uncorrectable Errors : 0<br />
BBU : Present<br />
BBU : Yes<br />
Cache When BBU Bad : Disabled<br />
</pre><br />
<br />
Confirm that the battery is present and error-free.<br />
<br />
If all that checks out, you're ready to proceed with the verify. After pressing enter, the verify is started and here's what you see:<br />
<pre><br />
Start Check Consistency on Virtual Drive 0 (target id: 0) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Start Check Consistency on Virtual Drive 1 (target id: 1) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Start Check Consistency on Virtual Drive 2 (target id: 2) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Check Consistency<br />
<br />
Progress of Virtual Drives...<br />
<br />
Virtual Drive # Percent Complete Time Elps<br />
0 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:03<br />
1 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:02<br />
2 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:01<br />
<br />
Press <ESC> key to quit...<br />
</pre><br />
<br />
The progress for each drive is displayed until all drives have completed the verify. We just want to make sure that each drive goes to completion. No followup is needed...though there probably is a log or history where we can get more info.<br />
<br />
You will notice that jail7 does not run a verify- that's on purpose. The last time we tried this it crashed the system. So, this must be run from the BIOS (take the system offline for a couple hours).<br />
<br />
See [[RAIC_CLI#LSI|LSI RAID CLI Reference]] for more details on how to use the CLI<br />
<br />
=== LSI-based controllers (megaraid) ===<br />
<br />
There is a CLI for this however it's easier to do this with a curses GUI app: megaraid<br />
<br />
Currently only on these servers: virt15, virt16, and firewall2<br />
<br />
To run:<br />
<br />
# cd /usr/local/sbin/; megamgr<br />
<br />
Main menu:<br />
<pre>²ÚÄÄManagement MenuÄÄ¿²<br />
²³ Configure ³²<br />
²³ Initialize ³²<br />
²³ Objects ³²<br />
²³ Rebuild ³²<br />
²³ Check Consistency ³²<br />
²³ Advanced Menu ³²<br />
²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²</pre><br />
<br />
Before you check consistency, make sure the arrays are healthy.<br />
<br />
Objects -> Physical Drive<br />
<br />
Then look to make sure they're all <tt>ONLIN</tt><br />
<pre><br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄObjects - PHYSICAL DRIVE SELECTION MENUÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²<br />
²²²²²²²²³ ³²²²²²²<br />
²²²²²²²²³ Channel-1 ³²²²²²²<br />
²ÚÄÄMana³ ID ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»x ³²²²²²²<br />
²³ Confi³ 0º* ONLIN A01-01º ³²²²²²²<br />
²³ Initi³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²³ Objec³ 1º* ONLIN A01-02º ³²²²²²²<br />
²³ Rebui³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²³ Check³ 2º* ONLIN A02-01º ³²²²²²²<br />
²³ Advan³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²ÀÄÄÄÄÄij 3º* ONLIN A02-02º ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 4º* ONLIN A03-01º ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 5º* ONLIN A03-02ºþ ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 6º* º ³²²²²²²<br />
²²²²²²²²³ ÈÍÍÍÍÍÍÍÍÍÍÍÍÍͼx ³²²²²²²<br />
²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²<br />
²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²²²<br />
²²²²²²²²³Ch-1 ID-5 DISK 140013MB SEAGATE ST3146707LC 0003 ³²²²²²²²²<br />
²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²<br />
</pre><br />
<br />
<br />
Once that's done, hit escape once then the back arror to move back to the Objects menu.<br />
So you select Objects -> Logical Drive -> Logical Drive 1 -> Check Consistency -> YES<br />
<br />
<pre><br />
<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÚÄLogical Drives(02)Ä¿²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ Logical Drive 1 ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²ÚÄÄÄÄObjectsÄÄij Logical Drive 2 ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²ÚÄÄManagemen³ Adapter ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Configure ³ Logical Drive ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Initialize³ Physical Drive ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Objects ³ Channel ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Rebuild ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Check Consistency ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Advanced Menu ³²²²²²²²ÚÄÄÄÄLogical Drive 1ÄÄÄÄÄ¿²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²³ Initialize ÚÄCheck Consistency-1 ?Ä¿²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ Check Consiste³ YES ³²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ View/Update Pa³ NO ³²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³Select YES Or NO³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
</pre><br />
<br />
Then watch the progress. When done, escape back to Logical Drive then repeat for Logical Drive 2. If you ^C or accidentally escape out, you can come back in running the same commands and watch the progress again (it won't restart).<br />
<br />
You can exit megamgr by escaping out or ^C<br />
<br />
=== 3ware ===<br />
<br />
We are using 3ware controllers on backup1 & backup2. Running the verify script will give you different output based on the type of controller:<br />
<br />
<pre><br />
backup2 /d2# sh /root/verify.sh<br />
Controller: c0<br />
-------------<br />
Driver: 1.50.01.002<br />
Model: 7500-8<br />
FW: FE7X 1.05.00.068<br />
BIOS: BE7X 1.08.00.048<br />
Monitor: ME7X 1.01.00.040<br />
Serial #: F11605A3180172<br />
PCB: Rev3<br />
PCHIP: 1.30-33<br />
ACHIP: 3.20<br />
<br />
<br />
# of units: 3<br />
Unit 0: JBOD 186.31 GB ( 390721968 blocks): OK<br />
Unit 1: RAID 5 465.77 GB ( 976790016 blocks): DEGRADED<br />
Unit 5: RAID 5 698.65 GB ( 1465185024 blocks): DEGRADED<br />
<br />
# of ports: 8<br />
Port 0: WDC WD2000JB-00KFA0 WD-WCAMT1451690 186.31 GB (390721968 blocks): OK(unit 0)<br />
Port 1: WDC WD2500JB-00GVC0 WD-WCAL78219488 232.88 GB (488397168 blocks): OK(unit 1)<br />
Port 2: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT)<br />
Port 3: WDC WD2500JB-00GVC0 WD-WMAL73882417 232.88 GB (488397168 blocks): OK(unit 1)<br />
Port 4: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT)<br />
Port 5: WDC WD2500JB-00GVA0 WD-WMAL71338097 232.88 GB (488397168 blocks): OK(unit 5)<br />
Port 6: WDC WD2500JB-32EVA0 WD-WMAEH1301595 232.88 GB (488397168 blocks): OK(unit 5)<br />
Port 7: WDC WD2500JB-00GVC0 WD-WCAL78165566 232.88 GB (488397168 blocks): OK(unit 5)<br />
Controller: c1<br />
-------------<br />
Driver: 1.50.01.002<br />
Model: 7500-8<br />
FW: FE7X 1.05.00.068<br />
BIOS: BE7X 1.08.00.048<br />
Monitor: ME7X 1.01.00.040<br />
Serial #: F11605A3180167<br />
PCB: Rev3<br />
PCHIP: 1.30-33<br />
ACHIP: 3.20<br />
<br />
<br />
# of units: 2<br />
Unit 0: RAID 5 698.65 GB ( 1465185024 blocks): OK<br />
Unit 4: RAID 5 698.65 GB ( 1465185024 blocks): OK<br />
<br />
# of ports: 8<br />
Port 0: WDC WD2500JB-00GVA0 WD-WMAL71301258 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 1: WDC WD2500JB-00GVA0 WD-WMAL71322705 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 2: WDC WD2500JB-00GVA0 WD-WMAL71945050 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 3: WDC WD2500JB-00GVA0 WD-WMAL71316201 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 4: WDC WD2500JB-00GVC0 WD-WCAL78323749 232.88 GB (488397168 blocks): OK(unit 4)<br />
Port 5: WDC WD3200AAJB-00J3A0 WD-WCAV2V689068 298.09 GB (625142448 blocks): OK(unit 4)<br />
Port 6: WDC WD2500JB-00GVC0 WD-WCAL78234420 232.88 GB (488397168 blocks): OK(unit 4)<br />
Port 7: WDC WD2500JB-00GVC0 WD-WCAL78592213 232.88 GB (488397168 blocks): OK(unit 4)<br />
backup2 /d2#<br />
</pre><br />
<br />
On backup2 look for all ok, no verify.<br />
<br />
<pre><br />
[root@backup3 ~]# sh /root/verify.sh<br />
/c2 Driver Version = 1.26.02.002<br />
/c2 Model = 8006-2LP<br />
/c2 Available Memory = 512KB<br />
/c2 Firmware Version = FE8S 1.05.00.068<br />
/c2 Bios Version = BE7X 1.08.00.048<br />
/c2 Boot Loader Version = ME7X 1.01.00.040<br />
/c2 Serial Number = L018501C6481395<br />
/c2 PCB Version = Rev5<br />
/c2 PCHIP Version = 1.30-66<br />
/c2 ACHIP Version = 3.20<br />
/c2 Total Optimal Units = 1<br />
/c2 Not Optimal Units = 0<br />
<br />
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u1 RAID-1 OK - - - 931.512 ON -<br />
<br />
Port Status Unit Size Blocks Serial<br />
---------------------------------------------------------------<br />
p0 OK u1 931.51 GB 1953525168 WD-WMAW31148820<br />
p1 OK u1 931.51 GB 1953525168 WD-WCATR0277515<br />
<br />
<br />
Ctl Date Severity Alarm Message<br />
------------------------------------------------------------------------------<br />
<br />
Sending start verify message to /c2/u1 ... Done.<br />
<br />
<br />
when done run:<br />
tw_cli /c2 show alarms<br />
<br />
[root@backup3 ~]#<br />
</pre><br />
<br />
Automatically starts the verify, just run <tt>tw_cli /c2 show alarms</tt> as instructed to see the results of the verify.<br />
<br />
<pre><br />
[root@backup1 /data/deprecated]# sh /root/verify.sh<br />
/c0 Driver Version = 2.26.02.010<br />
/c0 Model = 9650SE-8LPML<br />
/c0 Available Memory = 224MB<br />
/c0 Firmware Version = FE9X 4.06.00.004<br />
/c0 Bios Version = BE9X 4.05.00.015<br />
/c0 Boot Loader Version = BL9X 3.08.00.001<br />
/c0 Serial Number = L326025A8270177<br />
/c0 PCB Version = Rev 032<br />
/c0 PCHIP Version = 2.00<br />
/c0 ACHIP Version = 1.90<br />
/c0 Number of Ports = 8<br />
/c0 Number of Drives = 6<br />
/c0 Number of Units = 1<br />
/c0 Total Optimal Units = 1<br />
/c0 Not Optimal Units = 0<br />
/c0 JBOD Export Policy = off<br />
/c0 Disk Spinup Policy = 1<br />
/c0 Spinup Stagger Time Policy (sec) = 1<br />
/c0 Auto-Carving Policy = off<br />
/c0 Auto-Carving Size = 2048 GB<br />
/c0 Auto-Rebuild Policy = on<br />
/c0 Controller Bus Type = PCIe<br />
/c0 Controller Bus Width = 1 lane<br />
/c0 Controller Bus Speed = 2.5 Gbps/lane<br />
<br />
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u0 RAID-5 OK - - 64K 4656.56 ON ON<br />
<br />
Port Status Unit Size Blocks Serial<br />
---------------------------------------------------------------<br />
p0 OK u0 931.51 GB 1953525168 9QJ1Y017<br />
p1 DEVICE-ERROR u0 931.51 GB 1953525168 9QJ1ZN07<br />
p2 OK u0 931.51 GB 1953525168 9QJ2XK1R<br />
p3 OK u0 931.51 GB 1953525168 9QJ2010B<br />
p4 OK u0 1.36 TB 2930277168 6XW0L36T<br />
p5 OK u0 931.51 GB 1953525168 WD-WMATV2444836<br />
p6 NOT-PRESENT - - - -<br />
p7 NOT-PRESENT - - - -<br />
<br />
<br />
Ctl Date Severity Alarm Message<br />
------------------------------------------------------------------------------<br />
c0 [Sat May 12 11:27:15 2012] WARNING Sector repair completed: port=0, LBA=0x6AE571C<br />
c0 [Sat May 12 19:16:21 2012] WARNING Sector repair completed: port=1, LBA=0x40E62A23<br />
c0 [Sat May 12 21:40:56 2012] INFO Verify completed: unit=0<br />
c0 [Mon May 14 00:53:53 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFA7<br />
c0 [Mon May 14 00:58:21 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAA<br />
c0 [Mon May 14 04:35:13 2012] WARNING Sector repair completed: port=0, LBA=0x8FEF2CF<br />
c0 [Mon May 14 04:38:22 2012] WARNING Sector repair completed: port=0, LBA=0x8FEF2D1<br />
c0 [Tue May 15 22:53:46 2012] WARNING Sector repair completed: port=0, LBA=0x13C2622<br />
c0 [Wed May 16 00:39:31 2012] WARNING Sector repair completed: port=0, LBA=0x365A67F<br />
c0 [Wed May 16 00:39:37 2012] WARNING Sector repair completed: port=0, LBA=0x365A685<br />
c0 [Wed May 16 00:47:18 2012] WARNING Sector repair completed: port=0, LBA=0x365A687<br />
c0 [Sat May 19 00:01:44 2012] INFO Verify started: unit=0<br />
c0 [Sat May 19 04:46:20 2012] WARNING Sector repair completed: port=0, LBA=0x365A68E<br />
c0 [Sat May 19 13:37:06 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAC<br />
c0 [Sat May 19 13:37:28 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAE<br />
c0 [Sat May 19 13:37:47 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFB1<br />
c0 [Sat May 19 13:38:00 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFB3<br />
c0 [Sat May 19 21:47:45 2012] INFO Verify completed: unit=0<br />
c0 [Wed May 23 12:21:41 2012] INFO Cache synchronization completed: unit=0<br />
c0 [Fri May 25 00:08:19 2012] WARNING Sector repair completed: port=0, LBA=0x12DA76C<br />
c0 [Fri May 25 00:08:34 2012] WARNING Sector repair completed: port=0, LBA=0x12E4901<br />
c0 [Fri May 25 00:09:33 2012] WARNING Sector repair completed: port=0, LBA=0x12DA773<br />
c0 [Fri May 25 00:39:12 2012] WARNING Sector repair completed: port=0, LBA=0x42C597B<br />
c0 [Sat May 26 00:01:45 2012] INFO Verify started: unit=0<br />
c0 [Sat May 26 00:42:05 2012] WARNING Sector repair completed: port=1, LBA=0x323C1AC<br />
c0 [Sat May 26 00:51:43 2012] WARNING Sector repair completed: port=1, LBA=0x323C1AE<br />
c0 [Sat May 26 01:54:21 2012] WARNING Sector repair completed: port=1, LBA=0x2F0D302<br />
c0 [Sat May 26 02:06:38 2012] WARNING Sector repair completed: port=0, LBA=0x12DA777<br />
c0 [Sat May 26 02:07:21 2012] WARNING Sector repair completed: port=0, LBA=0x12E48FE<br />
c0 [Sat May 26 04:20:00 2012] WARNING Sector repair completed: port=1, LBA=0x2F0D306<br />
c0 [Sat May 26 04:32:58 2012] WARNING Sector repair completed: port=1, LBA=0x323C1B1<br />
c0 [Sat May 26 04:33:21 2012] WARNING Sector repair completed: port=1, LBA=0x323C1B3<br />
c0 [Sat May 26 04:33:44 2012] WARNING Sector repair completed: port=1, LBA=0x323C1BA<br />
c0 [Sat May 26 05:24:07 2012] WARNING Sector repair completed: port=1, LBA=0x3F83862<br />
c0 [Sat May 26 05:25:09 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 06:08:13 2012] WARNING Sector repair completed: port=0, LBA=0x4CDC6A2<br />
c0 [Sat May 26 09:49:35 2012] WARNING Sector repair completed: port=1, LBA=0x6CACD4A<br />
c0 [Sat May 26 18:10:44 2012] WARNING Sector repair completed: port=1, LBA=0x18F425EA<br />
c0 [Sat May 26 19:45:40 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:22:52 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:15 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:22 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:35 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:41 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:49 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:57 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:24:02 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:54:41 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 22:00:30 2012] INFO Verify completed: unit=0<br />
c0 [Sat Jun 2 00:01:43 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 2 00:30:17 2012] WARNING Sector repair completed: port=0, LBA=0x2B911E4<br />
c0 [Sat Jun 2 00:50:57 2012] WARNING Sector repair completed: port=0, LBA=0x5A807CA6<br />
c0 [Sat Jun 2 04:13:13 2012] WARNING Sector repair completed: port=0, LBA=0x2D18291<br />
c0 [Sat Jun 2 04:13:35 2012] WARNING Sector repair completed: port=0, LBA=0x2D1829F<br />
c0 [Sat Jun 2 21:48:02 2012] INFO Verify completed: unit=0<br />
c0 [Mon Jun 4 04:40:34 2012] WARNING Sector repair completed: port=1, LBA=0x4AF8098F<br />
c0 [Tue Jun 5 00:28:19 2012] WARNING Sector repair completed: port=1, LBA=0x263C5CD<br />
c0 [Tue Jun 5 00:33:06 2012] WARNING Sector repair completed: port=1, LBA=0x263C5CF<br />
c0 [Thu Jun 7 00:34:27 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B5F<br />
c0 [Thu Jun 7 00:38:50 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B61<br />
c0 [Fri Jun 8 00:07:13 2012] WARNING Sector repair completed: port=0, LBA=0xC131F6B<br />
c0 [Sat Jun 9 00:01:41 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 9 00:29:11 2012] WARNING Sector repair completed: port=0, LBA=0x6C7614D<br />
c0 [Sat Jun 9 00:38:25 2012] WARNING Sector repair completed: port=0, LBA=0x6C76152<br />
c0 [Sat Jun 9 04:02:30 2012] WARNING Sector repair completed: port=1, LBA=0x263C5D1<br />
c0 [Sat Jun 9 04:02:52 2012] WARNING Sector repair completed: port=1, LBA=0x263C5D3<br />
c0 [Sat Jun 9 04:07:32 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E12<br />
c0 [Sat Jun 9 04:07:57 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E15<br />
c0 [Sat Jun 9 04:08:16 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E17<br />
c0 [Sat Jun 9 04:08:45 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E19<br />
c0 [Sat Jun 9 04:15:04 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B64<br />
c0 [Sat Jun 9 04:15:26 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B66<br />
c0 [Sat Jun 9 04:15:45 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B68<br />
c0 [Sat Jun 9 04:15:59 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B6C<br />
c0 [Sat Jun 9 04:16:13 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B6E<br />
c0 [Sat Jun 9 21:48:52 2012] INFO Verify completed: unit=0<br />
c0 [Thu Jun 14 00:40:10 2012] WARNING Sector repair completed: port=0, LBA=0x334F14B<br />
c0 [Sat Jun 16 00:01:38 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 16 21:16:19 2012] INFO Verify completed: unit=0<br />
c0 [Tue Jun 19 02:03:43 2012] WARNING Sector repair completed: port=1, LBA=0xFE41EAD<br />
c0 [Wed Jun 20 02:30:02 2012] WARNING Sector repair completed: port=1, LBA=0xD99145C<br />
c0 [Sat Jun 23 00:01:36 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 23 04:27:04 2012] WARNING Sector repair completed: port=1, LBA=0x2FAD311<br />
c0 [Sat Jun 23 06:52:38 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC8D<br />
c0 [Sat Jun 23 06:53:03 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC91<br />
c0 [Sat Jun 23 06:53:21 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC94<br />
c0 [Sat Jun 23 17:00:22 2012] WARNING Sector repair completed: port=1, LBA=0xF9AC7C9<br />
c0 [Sat Jun 23 21:15:19 2012] INFO Verify completed: unit=0<br />
c0 [Sat Jun 30 00:01:34 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 30 05:24:13 2012] WARNING Sector repair completed: port=0, LBA=0x3FAA9E7<br />
c0 [Sat Jun 30 14:49:39 2012] WARNING Sector repair completed: port=1, LBA=0x869931C<br />
c0 [Sat Jun 30 21:31:05 2012] INFO Verify completed: unit=0<br />
c0 [Tue Jul 3 03:40:25 2012] WARNING Sector repair completed: port=1, LBA=0xD36C7F7<br />
c0 [Fri Jul 6 02:50:18 2012] WARNING Sector repair completed: port=1, LBA=0x3562470<br />
c0 [Fri Jul 6 22:18:26 2012] WARNING Sector repair completed: port=1, LBA=0x3563173<br />
c0 [Sat Jul 7 00:01:31 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 7 00:50:16 2012] WARNING Sector repair completed: port=0, LBA=0x76EE88<br />
c0 [Sat Jul 7 00:50:39 2012] WARNING Sector repair completed: port=0, LBA=0x76EE8F<br />
c0 [Sat Jul 7 21:39:36 2012] INFO Verify completed: unit=0<br />
c0 [Sun Jul 8 02:51:05 2012] WARNING Sector repair completed: port=0, LBA=0x67759D<br />
c0 [Sun Jul 8 02:53:55 2012] WARNING Sector repair completed: port=0, LBA=0x67759B<br />
c0 [Tue Jul 10 16:17:21 2012] WARNING Sector repair completed: port=0, LBA=0x15C8C695<br />
c0 [Wed Jul 11 22:51:22 2012] WARNING Sector repair completed: port=1, LBA=0x355BBD0<br />
c0 [Sat Jul 14 00:01:28 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 14 01:33:40 2012] WARNING Sector repair completed: port=1, LBA=0x1333BCF4<br />
c0 [Sat Jul 14 03:36:23 2012] WARNING Sector repair completed: port=1, LBA=0x2174773<br />
c0 [Sat Jul 14 11:26:44 2012] WARNING Sector repair completed: port=0, LBA=0x7429AB7<br />
c0 [Sat Jul 14 16:53:50 2012] WARNING Sector repair completed: port=1, LBA=0xA17EB3F<br />
c0 [Sat Jul 14 21:19:25 2012] INFO Verify completed: unit=0<br />
c0 [Wed Jul 18 05:08:47 2012] WARNING Sector repair completed: port=1, LBA=0x17D62EDC<br />
c0 [Wed Jul 18 05:14:15 2012] WARNING Sector repair completed: port=1, LBA=0x17D62EE1<br />
c0 [Thu Jul 19 03:24:59 2012] WARNING Sector repair completed: port=0, LBA=0x7733C3D<br />
c0 [Thu Jul 19 03:25:20 2012] WARNING Sector repair completed: port=0, LBA=0x773CEA5<br />
c0 [Thu Jul 19 03:28:16 2012] WARNING Sector repair completed: port=0, LBA=0x7733C42<br />
c0 [Thu Jul 19 03:28:41 2012] WARNING Sector repair completed: port=0, LBA=0x773CEAF<br />
c0 [Sat Jul 21 00:01:26 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 21 03:07:30 2012] WARNING Sector repair completed: port=1, LBA=0x1CC6936<br />
c0 [Sat Jul 21 03:07:52 2012] WARNING Sector repair completed: port=1, LBA=0x1CC6938<br />
c0 [Sat Jul 21 03:08:11 2012] WARNING Sector repair completed: port=1, LBA=0x1CC693A<br />
c0 [Sat Jul 21 16:43:56 2012] WARNING Sector repair completed: port=0, LBA=0xD04C914<br />
c0 [Sat Jul 21 16:45:31 2012] WARNING Sector repair completed: port=1, LBA=0xD456973<br />
c0 [Sat Jul 21 21:14:29 2012] INFO Verify completed: unit=0<br />
c0 [Wed Jul 25 03:37:25 2012] WARNING Sector repair completed: port=0, LBA=0x1F8E6C43<br />
c0 [Sat Jul 28 00:01:24 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 28 01:45:27 2012] WARNING Sector repair completed: port=0, LBA=0x11584AD<br />
c0 [Sat Jul 28 18:54:25 2012] WARNING Sector repair completed: port=1, LBA=0x447C3E6C<br />
c0 [Sat Jul 28 21:13:46 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 1 03:20:11 2012] WARNING Sector repair completed: port=0, LBA=0x805FEF<br />
c0 [Fri Aug 3 00:50:03 2012] WARNING Sector repair completed: port=0, LBA=0xCED0ACA<br />
c0 [Sat Aug 4 00:01:22 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 4 00:52:51 2012] WARNING Sector repair completed: port=0, LBA=0x805FF3<br />
c0 [Sat Aug 4 00:53:14 2012] WARNING Sector repair completed: port=0, LBA=0x805FF5<br />
c0 [Sat Aug 4 00:53:33 2012] WARNING Sector repair completed: port=0, LBA=0x805FF7<br />
c0 [Sat Aug 4 00:53:47 2012] WARNING Sector repair completed: port=0, LBA=0x805FF9<br />
c0 [Sat Aug 4 00:54:00 2012] WARNING Sector repair completed: port=0, LBA=0x805FFB<br />
c0 [Sat Aug 4 00:54:14 2012] WARNING Sector repair completed: port=0, LBA=0x805FFD<br />
c0 [Sat Aug 4 00:54:27 2012] WARNING Sector repair completed: port=0, LBA=0x805FFF<br />
c0 [Sat Aug 4 04:43:12 2012] WARNING Sector repair completed: port=1, LBA=0x16974289<br />
c0 [Sat Aug 4 04:58:17 2012] WARNING Sector repair completed: port=1, LBA=0x1697428E<br />
c0 [Sat Aug 4 20:54:53 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 8 03:21:55 2012] ERROR Drive timeout detected: port=1<br />
c0 [Wed Aug 8 15:31:44 2012] WARNING Sector repair completed: port=0, LBA=0x1A366CD3<br />
c0 [Sat Aug 11 00:01:21 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 11 20:40:51 2012] INFO Verify completed: unit=0<br />
c0 [Thu Aug 16 05:10:55 2012] WARNING Sector repair completed: port=0, LBA=0x1C22593<br />
c0 [Sat Aug 18 00:01:18 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 18 03:00:20 2012] WARNING Sector repair completed: port=0, LBA=0x1C225A5<br />
c0 [Sat Aug 18 03:43:00 2012] WARNING Sector repair completed: port=1, LBA=0x23EE91E<br />
c0 [Sat Aug 18 03:43:23 2012] WARNING Sector repair completed: port=1, LBA=0x23EE920<br />
c0 [Sat Aug 18 17:00:06 2012] WARNING Sector repair completed: port=1, LBA=0x137D066A<br />
c0 [Sat Aug 18 17:00:29 2012] WARNING Sector repair completed: port=1, LBA=0x137D066D<br />
c0 [Sat Aug 18 21:13:01 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 22 01:36:08 2012] WARNING Sector repair completed: port=0, LBA=0x2560A0F<br />
c0 [Wed Aug 22 01:37:42 2012] WARNING Sector repair completed: port=0, LBA=0x2560A13<br />
c0 [Fri Aug 24 04:01:36 2012] WARNING Sector repair completed: port=1, LBA=0x55C1A5DF<br />
c0 [Fri Aug 24 05:02:06 2012] WARNING Sector repair completed: port=1, LBA=0xCE3378A<br />
c0 [Sat Aug 25 00:01:17 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 25 00:31:06 2012] WARNING Sector repair completed: port=1, LBA=0x50F65D<br />
c0 [Sat Aug 25 00:39:52 2012] WARNING Sector repair completed: port=0, LBA=0x678FF4<br />
c0 [Sat Aug 25 03:43:15 2012] WARNING Sector repair completed: port=0, LBA=0x2560A15<br />
c0 [Sat Aug 25 03:43:39 2012] WARNING Sector repair completed: port=0, LBA=0x2560A19<br />
c0 [Sat Aug 25 03:43:58 2012] WARNING Sector repair completed: port=0, LBA=0x2560A1B<br />
c0 [Sat Aug 25 03:44:30 2012] WARNING Sector repair completed: port=0, LBA=0x2560A21<br />
c0 [Sat Aug 25 20:58:14 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 29 04:57:15 2012] WARNING Sector repair completed: port=1, LBA=0xF3957EB<br />
c0 [Sat Sep 1 00:01:15 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 1 03:21:52 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC86<br />
c0 [Sat Sep 1 03:22:15 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC88<br />
c0 [Sat Sep 1 03:22:34 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC8A<br />
c0 [Sat Sep 1 03:22:47 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC8C<br />
c0 [Sat Sep 1 17:17:22 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD1<br />
c0 [Sat Sep 1 17:17:45 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD3<br />
c0 [Sat Sep 1 17:18:04 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD5<br />
c0 [Sat Sep 1 21:36:56 2012] INFO Verify completed: unit=0<br />
c0 [Thu Sep 6 00:07:30 2012] WARNING Sector repair completed: port=0, LBA=0xDA3C64B<br />
c0 [Thu Sep 6 00:32:56 2012] WARNING Sector repair completed: port=1, LBA=0x6BBA816<br />
c0 [Sat Sep 8 00:01:13 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 8 00:09:56 2012] WARNING Sector repair completed: port=0, LBA=0xDEBC958<br />
c0 [Sat Sep 8 04:38:45 2012] WARNING Sector repair completed: port=0, LBA=0x38D254F<br />
c0 [Sat Sep 8 20:44:50 2012] INFO Verify completed: unit=0<br />
c0 [Mon Sep 10 01:26:34 2012] WARNING Sector repair completed: port=1, LBA=0xFFD8D5E<br />
c0 [Wed Sep 12 00:33:48 2012] WARNING Sector repair completed: port=1, LBA=0xE8DB928<br />
c0 [Wed Sep 12 00:36:33 2012] WARNING Sector repair completed: port=1, LBA=0x6D49411<br />
c0 [Fri Sep 14 01:59:39 2012] WARNING Sector repair completed: port=0, LBA=0x1467F1C<br />
c0 [Fri Sep 14 02:08:27 2012] WARNING Sector repair completed: port=1, LBA=0x14C8ABD<br />
c0 [Fri Sep 14 03:54:47 2012] WARNING Sector repair completed: port=0, LBA=0x1580C915<br />
c0 [Sat Sep 15 00:01:11 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 15 02:38:14 2012] WARNING Sector repair completed: port=0, LBA=0x1C178973<br />
c0 [Sat Sep 15 02:59:02 2012] WARNING Sector repair completed: port=0, LBA=0x1C178975<br />
c0 [Sat Sep 15 04:47:08 2012] WARNING Sector repair completed: port=0, LBA=0x3FA0356<br />
c0 [Sat Sep 15 04:47:31 2012] WARNING Sector repair completed: port=0, LBA=0x3FA0359<br />
c0 [Sat Sep 15 10:41:59 2012] WARNING Sector repair completed: port=0, LBA=0x6DFD1EC<br />
c0 [Sat Sep 15 13:25:23 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD100<br />
c0 [Sat Sep 15 13:25:31 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD104<br />
c0 [Sat Sep 15 13:25:54 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD106<br />
c0 [Sat Sep 15 17:10:50 2012] WARNING Sector repair completed: port=0, LBA=0x1C178977<br />
c0 [Sat Sep 15 20:59:57 2012] INFO Verify completed: unit=0<br />
c0 [Tue Sep 18 01:17:18 2012] WARNING Sector repair completed: port=1, LBA=0x803B05B<br />
c0 [Sat Sep 22 00:01:10 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 22 20:54:31 2012] INFO Verify completed: unit=0<br />
c0 [Tue Sep 25 01:56:47 2012] WARNING Sector repair completed: port=0, LBA=0x26E3909<br />
c0 [Sat Sep 29 00:01:08 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 29 02:04:14 2012] WARNING Sector repair completed: port=0, LBA=0x146AC03<br />
c0 [Sat Sep 29 10:58:39 2012] WARNING Sector repair completed: port=0, LBA=0x6D4EB0E<br />
c0 [Sat Sep 29 10:59:02 2012] WARNING Sector repair completed: port=0, LBA=0x6D4EB14<br />
c0 [Sat Sep 29 11:22:44 2012] WARNING Sector repair completed: port=0, LBA=0x6F79623<br />
c0 [Sat Sep 29 13:50:48 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D65E<br />
c0 [Sat Sep 29 13:51:11 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D661<br />
c0 [Sat Sep 29 13:51:30 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D663<br />
c0 [Sat Sep 29 20:57:34 2012] INFO Verify completed: unit=0<br />
c0 [Mon Oct 1 04:47:24 2012] WARNING Sector repair completed: port=0, LBA=0xC5BC6F2<br />
c0 [Tue Oct 2 02:00:27 2012] WARNING Sector repair completed: port=0, LBA=0x1547667<br />
c0 [Tue Oct 2 02:01:56 2012] WARNING Sector repair completed: port=0, LBA=0x154766F<br />
c0 [Tue Oct 2 05:02:31 2012] WARNING Sector repair completed: port=1, LBA=0xD67D054<br />
c0 [Tue Oct 2 05:04:14 2012] WARNING Sector repair completed: port=1, LBA=0xD67D056<br />
c0 [Wed Oct 3 01:22:12 2012] WARNING Sector repair completed: port=1, LBA=0x12AAF8CA<br />
c0 [Thu Oct 4 04:29:22 2012] WARNING Sector repair completed: port=0, LBA=0x13E6F992<br />
c0 [Thu Oct 4 05:10:51 2012] WARNING Sector repair completed: port=0, LBA=0x1C252A4<br />
c0 [Sat Oct 6 00:01:07 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 6 19:41:18 2012] WARNING Sector repair completed: port=1, LBA=0x5A5C3AE8<br />
c0 [Sat Oct 6 21:01:05 2012] INFO Verify completed: unit=0<br />
c0 [Mon Oct 8 00:32:06 2012] WARNING Sector repair completed: port=0, LBA=0x6C60D3E<br />
c0 [Tue Oct 9 03:51:03 2012] WARNING Sector repair completed: port=1, LBA=0x89B5EC9<br />
c0 [Thu Oct 11 04:21:17 2012] WARNING Sector repair completed: port=1, LBA=0x13F85833<br />
c0 [Sat Oct 13 00:01:05 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 13 05:12:40 2012] WARNING Sector repair completed: port=0, LBA=0x3FA5134<br />
c0 [Sat Oct 13 21:08:35 2012] INFO Verify completed: unit=0<br />
c0 [Tue Oct 16 03:53:50 2012] WARNING Sector repair completed: port=1, LBA=0x148AA1BD<br />
c0 [Thu Oct 18 03:20:30 2012] WARNING Sector repair completed: port=0, LBA=0x1C8DABCB<br />
c0 [Thu Oct 18 04:52:50 2012] WARNING Sector repair completed: port=0, LBA=0xE879057<br />
c0 [Sat Oct 20 00:01:04 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 20 02:19:25 2012] WARNING Sector repair completed: port=1, LBA=0x174B012<br />
c0 [Sat Oct 20 03:41:38 2012] WARNING Sector repair completed: port=0, LBA=0x256D93B<br />
c0 [Sat Oct 20 03:42:01 2012] WARNING Sector repair completed: port=0, LBA=0x256D93D<br />
c0 [Sat Oct 20 03:42:40 2012] WARNING Sector repair completed: port=0, LBA=0x256D940<br />
c0 [Sat Oct 20 03:42:59 2012] WARNING Sector repair completed: port=0, LBA=0x256D942<br />
c0 [Sat Oct 20 03:43:12 2012] WARNING Sector repair completed: port=0, LBA=0x256D944<br />
c0 [Sat Oct 20 03:43:26 2012] WARNING Sector repair completed: port=0, LBA=0x256D948<br />
c0 [Sat Oct 20 16:37:52 2012] WARNING Sector repair completed: port=0, LBA=0xE879060<br />
c0 [Sat Oct 20 16:38:15 2012] WARNING Sector repair completed: port=0, LBA=0xE879062<br />
c0 [Sat Oct 20 21:00:18 2012] INFO Verify completed: unit=0<br />
c0 [Sat Oct 20 23:49:01 2012] WARNING Sector repair completed: port=1, LBA=0x4473E908<br />
c0 [Sun Oct 21 03:42:26 2012] WARNING Sector repair completed: port=1, LBA=0x175BADD5<br />
c0 [Tue Oct 23 01:09:04 2012] WARNING Sector repair completed: port=1, LBA=0x6E524860<br />
c0 [Fri Oct 26 03:21:25 2012] WARNING Sector repair completed: port=0, LBA=0x802C61<br />
c0 [Fri Oct 26 04:22:21 2012] WARNING Sector repair completed: port=0, LBA=0x176353CD<br />
c0 [Sat Oct 27 00:01:03 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 27 00:49:35 2012] WARNING Sector repair completed: port=0, LBA=0x802C65<br />
c0 [Sat Oct 27 17:02:24 2012] WARNING Sector repair completed: port=1, LBA=0xC1FF26D<br />
c0 [Sat Oct 27 17:09:06 2012] WARNING Sector repair completed: port=0, LBA=0xDF621AD<br />
c0 [Sat Oct 27 21:30:57 2012] INFO Verify completed: unit=0<br />
c0 [Tue Oct 30 00:20:46 2012] WARNING Sector repair completed: port=0, LBA=0xE9FE2AB<br />
c0 [Wed Oct 31 02:02:03 2012] WARNING Sector repair completed: port=0, LBA=0x1460C25<br />
c0 [Wed Oct 31 02:04:05 2012] WARNING Sector repair completed: port=0, LBA=0x1460C28<br />
c0 [Thu Nov 1 00:48:34 2012] WARNING Sector repair completed: port=1, LBA=0xA7C92BE<br />
c0 [Thu Nov 1 05:04:45 2012] WARNING Sector repair completed: port=0, LBA=0x1C252C2<br />
<br />
[root@backup1 /data/deprecated]#<br />
</pre><br />
<br />
Look for failed drives and errors. Obviously from the above we need to probably replace drives 0 and 1 and drive 1 is even showing as having problems, yet the RAID array is healthy, amazingly. You also see the automatic verifies.<br />
<br />
Note: when rebuilding a degraded mirror, you will see no progress as it rebuilds in the cli<br />
<pre>Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u0 RAID-5 REBUILDING 0 - 64K 4656.56 OFF ON</pre><br />
<br />
<br />
=== areca ===<br />
<br />
We are using an areca controller on backup3. <br />
<br />
<pre><br />
[root@newbackup3 ~]# sh /root/verify.sh<br />
# Name Raid Name Level Capacity Ch/Id/Lun State<br />
===============================================================================<br />
1 ARC-1160-VOL#00 Raid Set # 00 Raid5 5000.0GB 00/00/00 Checking(19.7%)<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
# Name Disks TotalCap FreeCap DiskChannels State<br />
===============================================================================<br />
1 Raid Set # 00 6 6000.0GB 0.0GB 123456 Checking<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
Date-Time Device Event Type Elapsed Time Errors<br />
===============================================================================<br />
2012-12-05 20:40:58 ARC-1160-VOL#00 Start Checking<br />
2012-12-01 05:06:04 ARC-1160-VOL#00 Complete Init 027:30:45<br />
2012-11-30 01:35:19 ARC-1160-VOL#00 Start Initialize<br />
2026-08-06 01:34:52 H/W MONITOR Raid Powered On<br />
2012-11-30 01:33:36 ARC-1160-VOL#00 Stop Initialization 000:31:48<br />
2012-11-30 01:01:47 ARC-1160-VOL#00 Start Initialize<br />
2026-08-06 00:58:13 H/W MONITOR Raid Powered On<br />
2012-11-30 00:57:26 ARC-1160-VOL#00 Stop Initialization 000:57:07<br />
2012-11-30 00:00:19 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 23:56:48 H/W MONITOR Raid Powered On<br />
2026-08-05 23:52:58 H/W MONITOR Raid Powered On<br />
2026-08-05 23:50:14 H/W MONITOR Raid Powered On<br />
2026-08-05 23:43:30 H/W MONITOR Raid Powered On<br />
2012-11-29 23:10:07 ARC-1160-VOL#00 Stop Initialization 000:00:56<br />
2012-11-29 23:09:11 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 23:08:57 H/W MONITOR Raid Powered On<br />
2012-11-29 23:08:10 ARC-1160-VOL#00 Stop Initialization 000:20:41<br />
2012-11-29 22:47:29 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 22:46:59 H/W MONITOR Raid Powered On<br />
2026-08-05 22:45:55 H/W MONITOR Raid Powered On<br />
2026-08-05 22:44:53 H/W MONITOR Raid Powered On<br />
2026-08-05 22:42:06 H/W MONITOR Raid Powered On<br />
2026-08-05 22:40:50 H/W MONITOR Raid Powered On<br />
2012-11-29 22:40:04 ARC-1160-VOL#00 Stop Initialization 000:24:25<br />
2012-11-29 22:15:38 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 22:15:11 000:000001215B00 Restart Init LBA Point<br />
2026-08-05 22:15:10 H/W MONITOR Raid Powered On<br />
2012-11-29 21:56:38 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 21:56:12 H/W MONITOR Raid Powered On<br />
2026-08-05 21:56:04 IDE Channel #03 Device Inserted<br />
2012-11-29 21:55:13 IDE Channel #04 Device Inserted<br />
2012-11-29 21:55:03 IDE Channel #02 Device Inserted<br />
2026-08-05 21:53:09 H/W MONITOR Raid Powered On<br />
2026-08-05 20:51:46 H/W MONITOR Raid Powered On<br />
2026-08-05 20:49:56 H/W MONITOR Raid Powered On<br />
2026-08-05 20:48:29 H/W MONITOR Raid Powered On<br />
2026-08-05 20:46:29 H/W MONITOR Raid Powered On<br />
2026-08-05 20:44:49 H/W MONITOR Raid Powered On<br />
2026-08-05 20:43:01 H/W MONITOR Raid Powered On<br />
2026-08-05 20:36:25 H/W MONITOR Raid Powered On<br />
2026-08-05 20:31:18 H/W MONITOR Raid Powered On<br />
2026-08-05 20:30:08 H/W MONITOR Raid Powered On<br />
2026-08-05 20:08:40 H/W MONITOR Raid Powered On<br />
2026-08-05 20:06:11 H/W MONITOR Raid Powered On<br />
2026-08-05 20:05:14 H/W MONITOR Raid Powered On<br />
2026-08-05 20:03:58 H/W MONITOR Raid Powered On<br />
2026-08-05 20:00:56 H/W MONITOR Raid Powered On<br />
2026-08-05 19:57:57 H/W MONITOR Raid Powered On<br />
2026-08-05 19:56:15 H/W MONITOR Raid Powered On<br />
2026-08-05 19:55:05 H/W MONITOR Raid Powered On<br />
2026-08-05 17:24:36 H/W MONITOR Raid Powered On<br />
2026-08-05 17:22:43 H/W MONITOR Raid Powered On<br />
2026-08-05 04:50:42 H/W MONITOR Raid Powered On<br />
2026-08-05 04:47:33 H/W MONITOR Raid Powered On<br />
2026-08-05 04:43:57 H/W MONITOR Raid Powered On<br />
2026-08-05 04:18:52 H/W MONITOR Raid Powered On<br />
2026-08-05 04:17:30 H/W MONITOR Raid Powered On<br />
2026-08-05 04:13:30 H/W MONITOR Raid Powered On<br />
2026-08-05 04:10:26 H/W MONITOR Raid Powered On<br />
2026-08-05 04:09:23 H/W MONITOR Raid Powered On<br />
2026-08-05 00:08:09 H/W MONITOR Raid Powered On<br />
2026-08-05 00:07:12 H/W MONITOR Raid Powered On<br />
2026-08-05 00:05:51 H/W MONITOR Raid Powered On<br />
2026-08-05 00:04:27 H/W MONITOR Raid Powered On<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
press enter when ready to run verify<br />
</pre><br />
<br />
Look for failed drives and errors. <br />
<br />
When it proceed's to verifying, you can confirm with:<br />
<br />
<pre><br />
[root@newbackup3 ~]# cli64 vsf info<br />
# Name Raid Name Level Capacity Ch/Id/Lun State<br />
===============================================================================<br />
1 ARC-1160-VOL#00 Raid Set # 00 Raid5 5000.0GB 00/00/00 Checking(22.5%)<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
[root@newbackup3 ~]#<br />
</pre><br />
<br />
== Update OS list ==<br />
<br />
# check for any new VZ templates we want to offer: <tt>vzup2date -z</tt><br />
# see if there's any OS's we want to include in our colo install list. Update 2 places: <tt>signup/html/colo_quote.html</tt> & <tt>signup/html/step1.html</tt><br />
# update the mgmt database (ref_templates table, ref_systems table).<br />
<br />
= Infrequent tasks =<br />
<br />
== Free up space on gateway ==<br />
<br />
<pre>newgateway /var/spool# cd clientmqueue/<br />
newgateway /var/spool/clientmqueue# sh<br />
# for f in `ls`; do rm $f; done<br />
exit</pre><br />
<br />
== Free up space on mail ==<br />
<br />
You can clear out root mail:<br />
<pre><br />
mail /var/log# ll -h /var/mail/root<br />
-rw------- 1 root mail 543K Dec 19 13:05 /var/mail/root<br />
mail /var/log# rm /var/mail/root<br />
</pre><br />
<br />
Or you can archive mail logs:<br />
<pre><br />
mail /var/log# ls -l htt*<br />
-rw-r--r-- 1 root wheel 297436931 Dec 19 13:26 httpd-access.log<br />
-rw-r--r-- 1 root wheel 9824324 Jul 4 11:34 httpd-access.log.old.0.gz<br />
-rw-r--r-- 1 root wheel 6884137 Mar 17 2012 httpd-access.log.old.1.gz<br />
-rw-r--r-- 1 root wheel 18557444 Dec 3 2009 httpd-access.log.old.10.gz<br />
-rw-r--r-- 1 root wheel 14740263 Jan 9 2007 httpd-access.log.old.11.gz<br />
-rw-r--r-- 1 root wheel 14209465 Nov 28 2007 httpd-access.log.old.12.gz<br />
-rw-r--r-- 1 root wheel 16874396 Feb 19 2012 httpd-access.log.old.3.gz<br />
-rw-r--r-- 1 root wheel 14554859 Jul 22 2011 httpd-access.log.old.4.gz<br />
-rw-r--r-- 1 root wheel 10513227 Feb 18 2011 httpd-access.log.old.5.gz<br />
-rw-r--r-- 1 root wheel 7201946 Oct 29 2010 httpd-access.log.old.6.gz<br />
-rw-r--r-- 1 root wheel 10062537 May 6 2010 httpd-access.log.old.7.gz<br />
-rw-r--r-- 1 root wheel 10157042 Aug 12 2010 httpd-access.log.old.8.gz<br />
-rw-r--r-- 1 root wheel 11909534 Mar 4 2010 httpd-access.log.old.9.gz<br />
-rw-r--r-- 1 root wheel 59030930 Dec 19 13:01 httpd-error.log<br />
-rw-r--r-- 1 root wheel 3413134 Mar 4 2010 httpd-error.log.0.gz<br />
-rw-r--r-- 1 root wheel 795515 May 1 2007 httpd-error.log.1.gz<br />
-rw-r--r-- 1 root wheel 1142153 Nov 30 2007 httpd-error.log.2.gz<br />
-rw-r--r-- 1 root wheel 2325801 Feb 18 2011 httpd-error.log.gz<br />
<br />
mail /var/log# sh<br />
# for f in 12 11 10 9 8 7 6 5 4 3 2 1 0; do g=`echo $f+1|bc`; mv httpd-access.log.old.$f.gz httpd-access.log.old.$g.gz; done<br />
# mv httpd-access.log httpd-access.log.old.0<br />
# touch httpd-access.log<br />
# apachectl restart<br />
# gzip httpd-access.log.old.0<br />
<br />
# for f in 2 1 0; do g=`echo $f+1|bc`; mv httpd-error.log.$f.gz httpd-error.log.$g.gz; done<br />
# mv httpd-error.log httpd-error.log.0<br />
# touch httpd-error.log<br />
# apachectl restart<br />
# gzip httpd-error.log.0<br />
# exit<br />
</pre><br />
<br />
== Free up space on bwdb2 ==<br />
<br />
You can either remove items from <tt>/usr/home/archive</tt> or you can scp them to backup3:/data/bwdb2/archive .<br />
<br />
== Free up space on backup1 ==<br />
<br />
backup1 is our primary customer backup system. As usage grows over time, it needs to be regularly purged of old files. The easiest way to do this is by removing deprecated files. These mostly consist of cancelled customers or temporary dump/storage files (created during dump/restores). Our standard policy is to hang onto cancelled customers for 6mos after which we remove their files (as far as customers know their data is purged immediately, but we hang onto it just in case.. and in some cases we cancel a server due to non payment so this makes it easy to restore their system). To find files to remove:<br />
<br />
<pre>[root@backup1 ~]# cd /data/deprecated/<br />
[root@backup1 /data/deprecated]# ls<br />
2101-migrated-20120317.tgz old-683-cxld-20121021.tgz<br />
69.55.230.2-wwwbackup old-744-cxld-20120708.tgz<br />
991-DONTDELETE.tgz old-809-cxld-20120609.tgz<br />
archive-col02050-mdfile-cxld-20120409.gz old-854-cxld-20120621.tgz<br />
col01371.tgz old-931-cxld-20060513.tgz<br />
deleteme_ubuntu-10.10-x86_20111205 old-col00123-mdfile-noarchive-20120417.gz<br />
jail10_old old-col00147-vnfile-cxld-20120828.gz<br />
jail14_rsync_old old-col00419-dump-cxld-20120224.gz<br />
jail15_old old-col01098-vnfile-cxld-20120827.gz<br />
jail3_old old-col01278-dump-cxld-20120822<br />
jail4_old old-col01517-dump-cxld-20120828<br />
jail5_old old-col01669-dump-cxld-20120203.gz<br />
old-1009-cxld-20120608.tgz old-col01687-dump-cxld-20120909<br />
old-1012-cxld-20120411.tgz old-col01790-dump-cxld-20120828<br />
old-1052-cxld-20120721.tgz old-col01812-dump-cxld-20120820<br />
old-10631-cxld-20120622.tgz old-col01938-mdfile-cxld-20120619.gz<br />
old-10632-cxld-20120622.tgz old-col02095-mdfile-noarchive-20120523.gz<br />
old-10633-cxld-20120622.tgz olddebian-3.0-v15-20110610.tgz<br />
old-1236-cxld-20120621.tgz oldmod_frontpage-deb30-v15-20110610.tgz<br />
old-1381-cxld-20120404.tgz oldmod_perl-deb30-v15-20110610.tgz<br />
old-1422-cxld-20120721.tgz oldmod_ssl-deb30-v15-20110610.tgz<br />
old-14681-cxld-20120619.tgz oldmysql-deb30-v15-20110610.tgz<br />
old-1544-cxld-20120626.tgz oldproftpd-deb30-v15-20110610.tgz<br />
old-18351-cxld-20120605.tgz old_virt14<br />
old-1853-cxld-20120910.tgz old_virt18<br />
old-1963-cxld-20120206.tgz oldwebmin-deb30-v15-20110610.tgz<br />
old-1967-cxld-20120605.tgz suse.virt11.20120421.tgz<br />
old-1981-noarchive-20120729.tgz virt11<br />
old-2030-migrated-noarchive-20120727.tgz virt12_old<br />
old-2037-cxld-20120716.tgz virt13_old<br />
old-2065-cxld-20120727.tgz virt16_old<br />
old-2068-cxld-20120424.tgz virt4_old<br />
old-2085-cxld-20120531.tgz virt5_old<br />
old-364-cxld-20120904.tgz virt6_old<br />
old-446-cxld-20120512.tgz virt7_old<br />
old-613-cxld-20120601.tgz virt8_old<br />
[root@backup1 /data/deprecated]#<br />
</pre><br />
<br />
virtX_old and jailX_old are permanently archived, so ignore those as well as anything else marked not to delete or otherwise suspicious. Likewise, probably a good idea to try to hang onto oldTEMPLATE.gz as long as we can as well.<br />
Most of the stuff we want to delete is dated when it was deprecated, making this easy. So to remove files from 6 mos ago (running this in Oct):<br />
<br />
<pre>[root@backup1 /data/deprecated]# ls old*201204*<br />
old-1012-cxld-20120411.tgz old-2068-cxld-20120424.tgz<br />
old-1381-cxld-20120404.tgz old-col00123-mdfile-noarchive-20120417.gz<br />
[root@backup1 /data/deprecated]# rm old*201204*<br />
</pre><br />
<br />
Every few months you will also want to remove some of the snapshot archives for mail. We typically save the 1st, 10th, and 20th of each month. To do this you set aside the dates you want to save then remove months at a time, followed by restoring the set aside dates. Here's how that works:<br />
<pre>[root@backup1 /data/www/daily]# ls<br />
05 08-10-11 10-04-10 11-10-10 12-07-29 12-09-21 12-11-14<br />
06 08-10-21 10-04-20 11-10-20 12-07-30 12-09-22 12-11-15<br />
06-06-01-usr-home.tgz 08-11-01 10-05-01 11-11-01 12-07-31 12-09-23 12-11-16<br />
06-07-01-usr-home.tgz 08-11-10 10-05-11 11-11-10 12-08-01 12-09-24 12-11-17<br />
06-08-01-usr-home.tgz 08-11-20 10-05-20 11-11-20 12-08-02 12-09-25 12-11-18<br />
06-09-01-usr-home.tgz 08-12-01 10-06-01 11-12-01 12-08-03 12-09-26 12-11-19<br />
06-11-10 08-12-10 10-06-10 11-12-10 12-08-04 12-09-27 12-11-20<br />
06-12-21 08-12-20 10-06-20 11-12-20 12-08-05 12-09-28 12-11-21<br />
07-01-10 09-01-01 10-07-01 12-01-01 12-08-06 12-09-29 12-11-22<br />
07-01-20 09-01-10 10-07-10 12-01-10 12-08-07 12-09-30 12-11-23<br />
07-02-10 09-01-20 10-07-20 12-01-20 12-08-08 12-10-01 12-11-24<br />
07-02-20 09-02-01 10-08-01 12-02-01 12-08-09 12-10-02 12-11-25<br />
07-03-01 09-02-10 10-08-10 12-02-10 12-08-10 12-10-03 12-11-26<br />
07-03-20 09-02-20 10-08-20 12-02-20 12-08-11 12-10-04 12-11-27<br />
07-04-01 09-03-01 10-09-01 12-03-01 12-08-12 12-10-05 12-11-28<br />
07-04-10 09-03-10 10-09-10 12-03-10 12-08-13 12-10-06 12-11-29<br />
07-04-20 09-03-20 10-09-20 12-03-20 12-08-14 12-10-07 12-11-30<br />
07-05-01 09-04-01 10-10-01 12-04-01 12-08-15 12-10-08 12-12-01<br />
07-05-10 09-04-10 10-10-10 12-04-10 12-08-16 12-10-09 12-12-02<br />
07-05-20 09-04-20 10-10-20 12-04-20 12-08-17 12-10-10 12-12-03<br />
07-06-01 09-05-01 10-11-01 12-05-01 12-08-18 12-10-11 12-12-04<br />
07-06-10 09-05-10 10-11-10 12-05-10 12-08-19 12-10-12 12-12-05<br />
07-06-20 09-05-20 10-11-20 12-05-20 12-08-20 12-10-13 12-12-06<br />
07-07-20 09-06-01 10-12-01 12-06-01 12-08-21 12-10-14 12-12-07<br />
07-08-10 09-06-10 10-12-10 12-06-10 12-08-22 12-10-15 12-12-08<br />
07-08-20 09-06-20 10-12-20 12-06-20 12-08-23 12-10-16 12-12-09<br />
07-09-01 09-07-01 11-01-01 12-07-01 12-08-24 12-10-17 12-12-10<br />
07-10-01 09-07-10 11-01-10 12-07-02 12-08-25 12-10-18 12-12-11<br />
07-10-10 09-07-20 11-01-21 12-07-03 12-08-26 12-10-19 12-12-12<br />
07-10-20 09-08-01 11-02-01 12-07-04 12-08-27 12-10-20 12-12-13<br />
07-12-01 09-08-10 11-02-10 12-07-05 12-08-28 12-10-21 12-12-14<br />
07-12-10 09-08-20 11-02-20 12-07-06 12-08-29 12-10-22 12-12-15<br />
08-01-01 09-09-01 11-03-01 12-07-07 12-08-30 12-10-23 12-12-16<br />
08-01-20 09-09-10 11-03-10 12-07-08 12-08-31 12-10-24 12-12-17<br />
08-02-20 09-09-20 11-03-20 12-07-09 12-09-01 12-10-25 12-12-18<br />
08-03-01 09-10-01 11-04-01 12-07-10 12-09-02 12-10-26 12-12-19<br />
08-03-10 09-10-10 11-04-10 12-07-11 12-09-03 12-10-27 12-12-20<br />
08-03-20 09-10-20 11-04-20 12-07-12 12-09-04 12-10-28 12-12-21<br />
08-04-01 09-11-01 11-05-01 12-07-13 12-09-05 12-10-29 12-12-22<br />
08-04-20 09-11-10 11-05-10 12-07-14 12-09-06 12-10-30 12-12-23<br />
08-05-01 09-11-20 11-05-20 12-07-15 12-09-07 12-10-31 12-12-24<br />
08-05-10 09-12-01 11-06-01 12-07-16 12-09-08 12-11-01 12-12-25<br />
08-06-10 09-12-10 11-06-10 12-07-17 12-09-09 12-11-02 12-12-26<br />
08-06-20 09-12-20 11-06-20 12-07-18 12-09-10 12-11-03 12-12-27<br />
08-07-02 10-01-01 11-07-01 12-07-19 12-09-11 12-11-04 12-12-28<br />
08-07-10 10-01-10 11-07-10 12-07-20 12-09-12 12-11-05 2008-10-23<br />
08-07-20 10-01-20 11-07-20 12-07-21 12-09-13 12-11-06 bb.tgz<br />
08-08-01 10-02-01 11-08-01 12-07-22 12-09-14 12-11-07 boot<br />
08-08-10 10-02-10 11-08-10 12-07-23 12-09-15 12-11-08 current<br />
08-08-21 10-02-20 11-08-20 12-07-24 12-09-16 12-11-09 hold<br />
08-09-01 10-03-01 11-09-01 12-07-25 12-09-17 12-11-10<br />
08-09-10 10-03-10 11-09-10 12-07-26 12-09-18 12-11-11<br />
08-09-21 10-03-20 11-09-20 12-07-27 12-09-19 12-11-12<br />
08-10-01 10-04-01 11-10-01 12-07-28 12-09-20 12-11-13<br />
[root@backup1 /data/www/daily]#<br />
</pre><br />
<br />
So we see that everything up to July 2012 has been pruned. To prune July 2012 we do the following:<br />
<br />
<pre>mv 12-07-01 hold<br />
mv 12-07-10 hold<br />
mv 12-07-20 hold<br />
rm -fr 12-07*<br />
mv hold/* .</pre></div>75.80.128.147https://wiki.jcihosting.com/index.php?title=Routine_Maintenance&diff=2205Routine Maintenance2020-02-25T18:02:40Z<p>75.80.128.147: /* check NetHere */</p>
<hr />
<div>= Daily Tasks =<br />
<br />
== check load graphs == <br />
<br />
Click on the Load link in mgmt <br />
<br />
This screen shows you load levels on our servers and network traffic for critical machines (firewalls, backup servers).<br />
<br />
If you see load high or increasing <br />
<br />
FreeBSD: <br />
run [[VPS_Management#jtop|jtop]] (or [[VPS_Management#jt|jt]] > 7.x) and see if there are any runaway processes. Here are some examples of entries in top that are<br />
definitely runaway processes:<br />
<br />
<pre>79481 root 64 0 2256K 1056K CPU1 1 58:16 87.40% 87.40% nano<br />
50650 1000 64 0 1852K 1112K RUN 0 207.9H 84.08% 84.08% screen<br />
14829 www 2 0 39100K 31736K accept 0 104:24 46.54% 6.54% httpd<br />
42065 root 61 0 1300K 844K RUN 1 47.8H 91.36% 91.36% ee<br />
1328 www 56 0 18440K 10796K CPU1 0 64.4H 97.71% 97.71% httpd<br />
26251 user 57 0 6124K 1160K CPU1 1 82.9H 98.44% 98.44% screen<br />
89874 root 60 0 1352K 892K RUN 1 33.8H 65.82% 65.82% dialog<br />
38656 1000 64 0 3088K 2136K CPU0 0 806:13 97.95% 97.95% StutBot<br />
27630 root 64 0 1396K 972K RUN 1 76.8H 86.47% 86.47% ee</pre><br />
<br />
Linux:<br />
run [[VPS_Management#vwe|vwe]] to see which VPS’s have high loads. From there run <tt>[[VPS_Management#vp|vp]] <veid></tt> and/or <tt>[[VPS_Management#vt|vt]] <veid></tt> to see what's going on in that system.<br />
[[VPS_Management#vzstat|vzstat]] will also give you a nice picture of whats going on, systems with high numbers in the mlat column are likely culprits.<br />
<br />
examples of out of control procs:<br />
<br />
<pre>12183 nobody 16 0 4916 1348 1340 R 45.5 0.0 4249m httpd<br />
29266 #502 16 0 1852 796 792 R 22.5 0.0 1104m vim<br />
23860 #41 16 0 5472 5472 2076 R 98.9 0.2 31:41 python<br />
19227 bin 19 0 1688 716 652 R 99.9 0.0 321:08 wtrs_ui<br />
7762 apache 16 0 268 236 224 R 85.7 0.0 1010m ptrace<br />
4624 #501 20 0 4304 2400 2044 R 53.6 0.1 284:32 YoSucker<br />
20451 #506 20 0 1876 820 816 R 17.2 0.0 169:35 vim<br />
8834 #514 20 0 900 724 672 R 77.6 0.0 382:30 neostats<br />
31815 apache 14 0 3176 3176 1696 R 74.4 0.1 6:15 counter</pre><br />
<br />
Just kill -9 them and be done with it.<br />
<br />
Also, anytime you see `kmod` or `ptrace` - kill those immediaely no matter how much they are using - they are attempts to exploit the linux ptrace bug. They won't work, but they suck a lot of CPU...<br />
<br />
Also, any other processes that are at 90-100% cpu usage and have been running for any long period of time should be killed except for mysqld processes on FreeBSD. See above.<br />
<br />
However, there is an exception:<br />
<br />
if it is a mysqld, we don't want to kill their database. What you want to do is <tt>[[VPS_Management#jpid|jpid]] <pid></tt> to see who owns it, and then email them the paste containing the instructions for the nanny. Or you can simply do a <tt>kill -1 PID</tt> on the process to restart it.<br />
<br />
=== Load averages jump at night ===<br />
<br />
The load averages on the FreeBSD systems may jump up at night between 1 and 4 am - this is because the backups are running - if this is what is causing a jump in load, you will see processes like `rsync` in top eating a lot of CPU time.<br />
<br />
== check backups ==<br />
<br />
mgmt -> Motnroing -> Backups and make sure every machine was backed up the previous nite. <br />
Also look at df on backup1 and backup2 to make sure no disk is approaching full, though bb should warn us in advance. Please note - errors encountered when a backup script on any of the particular systems run will gnerate an email to support@johncompanies.com so you can know immediately the day after if the directory to be backed up has been moved or no longer exists. A paste exists for this to notify the customer of a non-existant file/dir.<br />
<br />
== check bb for warnings ==<br />
mgmt -> BigBrother<br />
<br />
Some events don't generate pages (on purpose). You will only see them by going to the bb main page.<br />
<br />
== check jail5 for crashed VPSs ==<br />
<br />
On jail5<br />
<br />
notrunning<br />
<br />
To restart a VPS <br />
<br />
vm restart col0XXXX<br />
<br />
== Check NetHere ==<br />
<br />
Check the NetHere servers<br />
<br />
=== Mail systems ===<br />
Check for possible SPAMMERs.<br />
<br />
==== Incoming ====<br />
Check mta-1 and mta-2 count of customer logins for possible<br />
customer SPAM compromises.<br />
<br />
login_count /logs/maillog | tail -30<br />
<br />
==== Outgoing ====<br />
<br />
Check outgoing queues on relay-1 and relay-2<br />
<br />
mail_count | tail -30<br />
<br />
To clean up outgoing queues of unwanted SPAM on relay-1 and relay-2.<br />
<br />
mail_cleanup [<sender's domain>]<br />
<br />
=== Nagios ===<br />
<br />
Check for other problems on NetHere servers<br />
<br />
https://nagios.nethere.net<br />
<br />
=== Cacti ===<br />
<br />
Check bandwidth usage on servers<br />
<br />
https://cacti.nethere.net<br />
<br />
= Monthly Tasks =<br />
== rotate pine sent mail (1st of month) ==<br />
On the 1st of the month, before any emails are sent out, quit out of pine, then log back in. Send mail from last month will be archived.<br />
If you mess up and do it on the 3rd (for example), you can go into the previous month's saved email and save emails from the current month into the <tt>sent-mail</tt> (current month) mailbox.<br />
<br />
== b/w caps ==<br />
On the 1st: remove any bwcaps put into the firewall (only really applies if a bwcap was added cause someone went over on b/w):<br />
<br />
ipfw list|grep pipe<br />
ipfw del [each rule listed]<br />
<br />
NOTE: this cronjob on newgateway will do some of that for you, provided you used one of the following pipe #s:<br />
0 0 1 * * /sbin/ipfw del 3 4 5 17331<br />
<br />
-------------------<br />
<br />
We really don’t do this anymore since we have centralized traffic accounting with netflow, but for posterity:<br />
<br />
Make sure all machines reset counters to 0 after midnight on the 1st<br />
Make sure they dumped a counter<br />
<br />
On each jail run:<br />
trafficgather.pl<br />
<br />
And on each virt:<br />
linuxtrafficgather.pl<br />
<br />
== Monthly RAID checks ==<br />
<br />
Every month we check the health of and verfy the parity on all our RAID-based systems.<br />
To facilitate this, we've created a simple script to start the process:<br />
<br />
sh /root/verify.sh<br />
<br />
=== Adaptec controllers ===<br />
Here's some sample output:<br />
<pre><br />
mail /usr/local/www/scripts# sh /root/verify.sh<br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> container list /f<br />
Executing: container list /full=TRUE<br />
Num Total Oth Chunk Scsi Partition<br />
Creation System<br />
Label Type Size Ctr Size Usage B:ID:L Offset:Size State RO Lk Task Done% Ent<br />
Date Time Files<br />
----- ------ ------ --- ------ ------- ------ ------------- ------- -- -- ------- ------ ---<br />
------ -------- ------<br />
0 Mirror 33.9GB Open 0:01:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:32<br />
/dev/aacd0 mirror0 0:00:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:32<br />
<br />
1 Mirror 33.9GB Open 0:02:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:50<br />
/dev/aacd1 mirror1 0:03:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:50<br />
<br />
<br />
AAC0> disk list /f<br />
Executing: disk list /full=TRUE<br />
<br />
B:ID:L Device Type Removable media Vendor-ID Product-ID Rev Blocks Bytes/Bl<br />
ock Usage Shared Rate<br />
------ -------------- --------------- --------- ---------------- ----- --------- --------<br />
--- ---------------- ------ ----<br />
0:00:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:01:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:02:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:03:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
<br />
AAC0> disk show smart<br />
Executing: disk show smart<br />
<br />
Smart Method of Enable<br />
Capable Informational Exception Performance Error<br />
B:ID:L Device Exceptions(MRIE) Control Enabled Count<br />
------ ------- ---------------- --------- ----------- ------<br />
0:00:0 Y 6 Y N 0<br />
0:01:0 Y 6 Y N 0<br />
0:02:0 Y 6 Y N 0<br />
0:03:0 Y 6 Y N 0<br />
0:06:0 N<br />
<br />
AAC0> task list<br />
Executing: task list<br />
<br />
Controller Tasks<br />
<br />
TaskId Function Done% Container State Specific1 Specific2<br />
------ -------- ------- --------- ----- --------- ---------<br />
<br />
No tasks currently running on controller<br />
<br />
AAC0> dia sh hi<br />
Executing: diagnostic show history<br />
No switches specified, defaulting to "/current".<br />
<br />
<br />
<br />
*** HISTORY BUFFER FROM CURRENT CONTROLLER RUN ***<br />
<br />
[00]: GetDiskLogEntry: container - 1, entry return 0<br />
[01]: Container 1 started SCRUB task<br />
[02]: Starting Mirror:1 scrub<br />
[03]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[04]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[05]: UpdateDiskLogIndex - Set - container 0, index 1<br />
[06]: GetDiskLogEntry: container - 0, entry return 1<br />
[07]: Container 0 started SCRUB task<br />
[08]: Starting Mirror:0 scrub<br />
[09]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[10]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[11]: Mirror Scrub Container:1 ErrorsFound:0<br />
[12]: Clear disk log: sector - 80, driveno 2<br />
[13]: Clear disk log: sector - 80, driveno 3<br />
[14]: Container 1 completed SCRUB task:<br />
[15]: Mirror Scrub Container:0 ErrorsFound:0<br />
[16]: Clear disk log: sector - 81, driveno 1<br />
[17]: Clear disk log: sector - 81, driveno 0<br />
[18]: Container 0 completed SCRUB task:<br />
[19]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[20]: GetDiskLogEntry: container - 0, entry return 0<br />
[21]: Container 0 started SCRUB task<br />
[22]: Starting Mirror:0 scrub<br />
[23]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[24]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[25]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[26]: GetDiskLogEntry: container - 1, entry return 1<br />
[27]: Container 1 started SCRUB task<br />
[28]: Starting Mirror:1 scrub<br />
[29]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[30]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[31]: Mirror Scrub Container:1 ErrorsFound:0<br />
[32]: Clear disk log: sector - 81, driveno 2<br />
[33]: Clear disk log: sector - 81, driveno 3<br />
[34]: Container 1 completed SCRUB task:<br />
[35]: Mirror Scrub Container:0 ErrorsFound:0<br />
[36]: Clear disk log: sector - 80, driveno 1<br />
[37]: Clear disk log: sector - 80, driveno 0<br />
[38]: Container 0 completed SCRUB task:<br />
[39]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[40]: GetDiskLogEntry: container - 0, entry return 0<br />
[41]: Container 0 started SCRUB task<br />
[42]: Starting Mirror:0 scrub<br />
[43]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[44]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[45]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[46]: GetDiskLogEntry: container - 1, entry return 1<br />
[47]: Container 1 started SCRUB task<br />
[48]: Starting Mirror:1 scrub<br />
[49]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[50]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[51]: Mirror Scrub Container:1 ErrorsFound:0<br />
[52]: Clear disk log: sector - 81, driveno 2<br />
[53]: Clear disk log: sector - 81, driveno 3<br />
[54]: Container 1 completed SCRUB task:<br />
[55]: Mirror Scrub Container:0 ErrorsFound:0<br />
[56]: Clear disk log: sector - 80, driveno 1<br />
[57]: Clear disk log: sector - 80, driveno 0<br />
[58]: Container 0 completed SCRUB task:<br />
[59]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[60]: GetDiskLogEntry: container - 0, entry return 0<br />
[61]: Container 0 started SCRUB task<br />
[62]: Starting Mirror:0 scrub<br />
[63]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[64]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[65]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[66]: GetDiskLogEntry: container - 1, entry return 1<br />
[67]: Container 1 started SCRUB task<br />
[68]: Starting Mirror:1 scrub<br />
[69]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[70]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[71]: Mirror Scrub Container:1 ErrorsFound:0<br />
[72]: Clear disk log: sector - 81, driveno 2<br />
[73]: Clear disk log: sector - 81, driveno 3<br />
[74]: Container 1 completed SCRUB task:<br />
[75]: Mirror Scrub Container:0 ErrorsFound:0<br />
[76]: Clear disk log: sector - 80, driveno 1<br />
[77]: Clear disk log: sector - 80, driveno 0<br />
[78]: Container 0 completed SCRUB task:<br />
[79]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[80]: GetDiskLogEntry: container - 0, entry return 0<br />
[81]: Container 0 started SCRUB task<br />
[82]: Starting Mirror:0 scrub<br />
[83]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[84]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[85]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[86]: GetDiskLogEntry: container - 1, entry return 1<br />
[87]: Container 1 started SCRUB task<br />
[88]: Starting Mirror:1 scrub<br />
[89]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[90]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[91]: Mirror Scrub Container:1 ErrorsFound:0<br />
[92]: Clear disk log: sector - 81, driveno 2<br />
[93]: Clear disk log: sector - 81, driveno 3<br />
[94]: Container 1 completed SCRUB task:<br />
[95]: Mirror Scrub Container:0 ErrorsFound:0<br />
[96]: Clear disk log: sector - 80, driveno 1<br />
[97]: Clear disk log: sector - 80, driveno 0<br />
[98]: Container 0 completed SCRUB task:<br />
[99]:<br />
<br />
========================<br />
History Output Complete.<br />
<br />
AAC0><br />
AAC0> exit<br />
Executing: exit<br />
<br />
press enter when ready to run verify <INS><br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> contai scr 0<br />
Executing: container scrub 0<br />
<br />
AAC0> contai scr 1<br />
Executing: container scrub 1<br />
<br />
AAC0> exit<br />
Executing: exit<br />
<br />
when done run: <br />
<br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
<br />
<br />
Nov 1 10:32:46 mail /kernel: aac0: **Monitor** Container 0 started SCRUB task<br />
Nov 1 10:32:47 mail /kernel: aac0: **Monitor** Container 1 started SCRUB task<br />
<br />
</pre><br />
<br />
Here's an analysis of what we're seeing and what we're looking for:<br />
<br />
<pre><br />
AAC0> container list /f<br />
Executing: container list /full=TRUE<br />
Num Total Oth Chunk Scsi Partition<br />
Creation System<br />
Label Type Size Ctr Size Usage B:ID:L Offset:Size State RO Lk Task Done% Ent<br />
Date Time Files<br />
----- ------ ------ --- ------ ------- ------ ------------- ------- -- -- ------- ------ ---<br />
------ -------- ------<br />
0 Mirror 33.9GB Open 0:01:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:32<br />
/dev/aacd0 mirror0 0:00:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:32<br />
<br />
1 Mirror 33.9GB Open 0:02:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:50<br />
/dev/aacd1 mirror1 0:03:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:50<br />
</pre><br />
<br />
This is showing you the health of the arrays. You're looking for ''Normal'' under the State column, and the absence of a ! in the sector size - sometimes, you'll see this:<br />
64.0KB!33.9GB <br />
<br />
That indicates a problem.<br />
<br />
<pre><br />
AAC0> disk show smart<br />
Executing: disk show smart<br />
<br />
Smart Method of Enable<br />
Capable Informational Exception Performance Error<br />
B:ID:L Device Exceptions(MRIE) Control Enabled Count<br />
------ ------- ---------------- --------- ----------- ------<br />
0:00:0 Y 6 Y N 0<br />
0:01:0 Y 6 Y N 0<br />
0:02:0 Y 6 Y N 0<br />
0:03:0 Y 6 Y N 0<br />
0:06:0 N<br />
</pre><br />
<br />
This shows you a SMART report output. Looking for values in the Error Count column.<br />
<pre><br />
AAC0> task list<br />
Executing: task list<br />
<br />
Controller Tasks<br />
<br />
TaskId Function Done% Container State Specific1 Specific2<br />
------ -------- ------- --------- ----- --------- ---------<br />
<br />
No tasks currently running on controller<br />
</pre><br />
<br />
Look for absence of tasks running- a bad thing would be to see a rebuild or verify running when you didn't initiate it.<br />
<br />
With the history output, you're looking for any anomalies or events since the last time a verify was run. If you see a drive with lots of problems, you may want to take backups before allowing the verify to run since it could replicate errors onto the good drive.<br />
<br />
After you see the history output, it will prompt you to press enter to run the verify. If you're happy with all the output you're seeing- mirror is healthy, history looks good, it's safe to proceed. Otherwise ^C to exit. After hitting enter it will start the verify and start to tail the messages log file (so you can easily see when the verify is complete). Here's what that'll look like:<br />
<br />
<pre><br />
Nov 1 14:38:08 mail /kernel: aac0: **Monitor** Container 1 completed SCRUB task:<br />
Nov 1 14:46:45 mail /kernel: aac0: **Monitor** Container 0 completed SCRUB task:<br />
</pre><br />
<br />
So, putting it all together, after hitting enter to start the verify, you'll see:<br />
<br />
<pre><br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> contai scr 0<br />
Executing: container scrub 0<br />
<br />
AAC0> contai scr 1<br />
Executing: container scrub 1<br />
<br />
AAC0> exit<br />
Executing: exit<br />
<br />
when done run: <br />
<br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
<br />
<br />
Nov 1 10:32:46 mail /kernel: aac0: **Monitor** Container 0 started SCRUB task<br />
Nov 1 10:32:47 mail /kernel: aac0: **Monitor** Container 1 started SCRUB task<br />
<br />
</pre><br />
<br />
When the scrub(s) (verify) are complete - if the server has multiple logical drives, it will run both in parallel - you should exit the tail of the log file (^C) and run:<br />
<br />
<pre><br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
</pre><br />
<br />
Which will show you the diagnostic history, you're looking for the results of the most recent scrub:<br />
<br />
<pre><br />
[100]: Mirror Scrub Container:1 ErrorsFound:0<br />
[101]: Clear disk log: sector - 81, driveno 2<br />
[102]: Clear disk log: sector - 81, driveno 3<br />
[103]: Container 1 completed SCRUB task:<br />
[104]: Mirror Scrub Container:0 ErrorsFound:0<br />
[105]: Clear disk log: sector - 80, driveno 1<br />
[106]: Clear disk log: sector - 80, driveno 0<br />
[107]: Container 0 completed SCRUB task:<br />
</pre><br />
<br />
^C to exit the RAID CLI.<br />
<br />
If you see:<br />
<br />
[104]: Mirror Scrub Container:0 ErrorsFound:5<br />
<br />
You'll want to rerun the verify on that drive till it shows 0, or perhaps replace the drive- you should be able to see from the output which drive had the problem.<br />
<br />
Depending on the size and how busy the drive is, the verify can take anywhere from an hour to the better part of a day.<br />
<br />
You will notice that the diagnostic history is not shown on our modern adaptec cards (i.e. any adaptec card not in a Dell 2450). The reason for this is the history is never cleared, so there's simply too much data to show and it just crashes the CLI. So, don't bother trying to see it...which does make it hard to see if there are problems going on, so you just need to watch the scrub to see it goes to 100%. You will also notice that on some servers there's no tail of messages. Again, this is cause no data is shown there about the completion of the scrub. The thing to do here is to go into the CLI and continue to <tt>show tasks</tt> to monitor scrub progress.<br />
<br />
See [[RAIC_CLI#Adaptec|Adaptec RAID CLI Reference]] for more details on how to use the CLI<br />
<br />
=== DELL (LSI-based) SAS controllers ===<br />
<br />
Here's what the output looks like when running verify.sh on a LSI-based card:<br />
<br />
<pre><br />
jail2 /mnt/data2# sh /root/verify.sh<br />
<br />
Adapter #0<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 0<br />
Device Id: 0<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018396142<br />
SAS Address(1): 0x0<br />
Connected Port Number: 0(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DESN<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 1<br />
Device Id: 1<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018395db2<br />
SAS Address(1): 0x0<br />
Connected Port Number: 1(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DERV<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 2<br />
Device Id: 2<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50006eece89<br />
SAS Address(1): 0x0<br />
Connected Port Number: 2(path0)<br />
Inquiry Data: SEAGATE ST3300555SS T2113LM4BFBZ<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 3<br />
Device Id: 3<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50006eee035<br />
SAS Address(1): 0x0<br />
Connected Port Number: 3(path0)<br />
Inquiry Data: SEAGATE ST3300555SS T2113LM4BGF7<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 4<br />
Device Id: 4<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50004bd7ea5<br />
SAS Address(1): 0x0<br />
Connected Port Number: 4(path0)<br />
Inquiry Data: SEAGATE ST3300656SS HS093QP0G8SW<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 5<br />
Device Id: 5<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e01f1c4112<br />
SAS Address(1): 0x0<br />
Connected Port Number: 5(path0)<br />
Inquiry Data: FUJITSU MBA3300RC D306BJ15P9201W06<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 0 (Target Id: 0)<br />
Name:<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:139392MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 1 (Target Id: 1)<br />
Name:MIRROR1<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 2 (Target Id: 2)<br />
Name:MIRROR2<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
Battery FRU : N/A<br />
Battery Warning : Enabled<br />
Memory Correctable Errors : 0<br />
Memory Uncorrectable Errors : 0<br />
BBU : Present<br />
BBU : Yes<br />
Cache When BBU Bad : Disabled<br />
press enter when ready to run verify<br />
<br />
</pre><br />
<br />
Before pressing enter, here's what we're looking for:<br />
<br />
<pre><br />
Enclosure Device ID: 32<br />
Slot Number: 0<br />
Device Id: 0<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018396142<br />
SAS Address(1): 0x0<br />
Connected Port Number: 0(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DESN<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
</pre><br />
<br />
This is the output shown for each physical drive in the system. We're looking to confirm it's Firmware state is ''Online'', and Media Error Count, Other Error Count, and Predictive Failure Count are all zero (or near zero).<br />
<br />
<pre><br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 1 (Target Id: 1)<br />
Name:MIRROR1<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
</pre><br />
<br />
This is the output for each logical drive. We're looking for State ''Optimal''. Also confirm <tt>Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU</tt><br />
<br />
<pre><br />
Exit Code: 0x00<br />
Battery FRU : N/A<br />
Battery Warning : Enabled<br />
Memory Correctable Errors : 0<br />
Memory Uncorrectable Errors : 0<br />
BBU : Present<br />
BBU : Yes<br />
Cache When BBU Bad : Disabled<br />
</pre><br />
<br />
Confirm that the battery is present and error-free.<br />
<br />
If all that checks out, you're ready to proceed with the verify. After pressing enter, the verify is started and here's what you see:<br />
<pre><br />
Start Check Consistency on Virtual Drive 0 (target id: 0) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Start Check Consistency on Virtual Drive 1 (target id: 1) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Start Check Consistency on Virtual Drive 2 (target id: 2) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Check Consistency<br />
<br />
Progress of Virtual Drives...<br />
<br />
Virtual Drive # Percent Complete Time Elps<br />
0 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:03<br />
1 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:02<br />
2 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:01<br />
<br />
Press <ESC> key to quit...<br />
</pre><br />
<br />
The progress for each drive is displayed until all drives have completed the verify. We just want to make sure that each drive goes to completion. No followup is needed...though there probably is a log or history where we can get more info.<br />
<br />
You will notice that jail7 does not run a verify- that's on purpose. The last time we tried this it crashed the system. So, this must be run from the BIOS (take the system offline for a couple hours).<br />
<br />
See [[RAIC_CLI#LSI|LSI RAID CLI Reference]] for more details on how to use the CLI<br />
<br />
=== LSI-based controllers (megaraid) ===<br />
<br />
There is a CLI for this however it's easier to do this with a curses GUI app: megaraid<br />
<br />
Currently only on these servers: virt15, virt16, and firewall2<br />
<br />
To run:<br />
<br />
# cd /usr/local/sbin/; megamgr<br />
<br />
Main menu:<br />
<pre>²ÚÄÄManagement MenuÄÄ¿²<br />
²³ Configure ³²<br />
²³ Initialize ³²<br />
²³ Objects ³²<br />
²³ Rebuild ³²<br />
²³ Check Consistency ³²<br />
²³ Advanced Menu ³²<br />
²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²</pre><br />
<br />
Before you check consistency, make sure the arrays are healthy.<br />
<br />
Objects -> Physical Drive<br />
<br />
Then look to make sure they're all <tt>ONLIN</tt><br />
<pre><br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄObjects - PHYSICAL DRIVE SELECTION MENUÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²<br />
²²²²²²²²³ ³²²²²²²<br />
²²²²²²²²³ Channel-1 ³²²²²²²<br />
²ÚÄÄMana³ ID ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»x ³²²²²²²<br />
²³ Confi³ 0º* ONLIN A01-01º ³²²²²²²<br />
²³ Initi³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²³ Objec³ 1º* ONLIN A01-02º ³²²²²²²<br />
²³ Rebui³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²³ Check³ 2º* ONLIN A02-01º ³²²²²²²<br />
²³ Advan³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²ÀÄÄÄÄÄij 3º* ONLIN A02-02º ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 4º* ONLIN A03-01º ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 5º* ONLIN A03-02ºþ ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 6º* º ³²²²²²²<br />
²²²²²²²²³ ÈÍÍÍÍÍÍÍÍÍÍÍÍÍͼx ³²²²²²²<br />
²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²<br />
²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²²²<br />
²²²²²²²²³Ch-1 ID-5 DISK 140013MB SEAGATE ST3146707LC 0003 ³²²²²²²²²<br />
²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²<br />
</pre><br />
<br />
<br />
Once that's done, hit escape once then the back arror to move back to the Objects menu.<br />
So you select Objects -> Logical Drive -> Logical Drive 1 -> Check Consistency -> YES<br />
<br />
<pre><br />
<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÚÄLogical Drives(02)Ä¿²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ Logical Drive 1 ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²ÚÄÄÄÄObjectsÄÄij Logical Drive 2 ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²ÚÄÄManagemen³ Adapter ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Configure ³ Logical Drive ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Initialize³ Physical Drive ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Objects ³ Channel ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Rebuild ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Check Consistency ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Advanced Menu ³²²²²²²²ÚÄÄÄÄLogical Drive 1ÄÄÄÄÄ¿²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²³ Initialize ÚÄCheck Consistency-1 ?Ä¿²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ Check Consiste³ YES ³²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ View/Update Pa³ NO ³²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³Select YES Or NO³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
</pre><br />
<br />
Then watch the progress. When done, escape back to Logical Drive then repeat for Logical Drive 2. If you ^C or accidentally escape out, you can come back in running the same commands and watch the progress again (it won't restart).<br />
<br />
You can exit megamgr by escaping out or ^C<br />
<br />
=== 3ware ===<br />
<br />
We are using 3ware controllers on backup1 & backup2. Running the verify script will give you different output based on the type of controller:<br />
<br />
<pre><br />
backup2 /d2# sh /root/verify.sh<br />
Controller: c0<br />
-------------<br />
Driver: 1.50.01.002<br />
Model: 7500-8<br />
FW: FE7X 1.05.00.068<br />
BIOS: BE7X 1.08.00.048<br />
Monitor: ME7X 1.01.00.040<br />
Serial #: F11605A3180172<br />
PCB: Rev3<br />
PCHIP: 1.30-33<br />
ACHIP: 3.20<br />
<br />
<br />
# of units: 3<br />
Unit 0: JBOD 186.31 GB ( 390721968 blocks): OK<br />
Unit 1: RAID 5 465.77 GB ( 976790016 blocks): DEGRADED<br />
Unit 5: RAID 5 698.65 GB ( 1465185024 blocks): DEGRADED<br />
<br />
# of ports: 8<br />
Port 0: WDC WD2000JB-00KFA0 WD-WCAMT1451690 186.31 GB (390721968 blocks): OK(unit 0)<br />
Port 1: WDC WD2500JB-00GVC0 WD-WCAL78219488 232.88 GB (488397168 blocks): OK(unit 1)<br />
Port 2: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT)<br />
Port 3: WDC WD2500JB-00GVC0 WD-WMAL73882417 232.88 GB (488397168 blocks): OK(unit 1)<br />
Port 4: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT)<br />
Port 5: WDC WD2500JB-00GVA0 WD-WMAL71338097 232.88 GB (488397168 blocks): OK(unit 5)<br />
Port 6: WDC WD2500JB-32EVA0 WD-WMAEH1301595 232.88 GB (488397168 blocks): OK(unit 5)<br />
Port 7: WDC WD2500JB-00GVC0 WD-WCAL78165566 232.88 GB (488397168 blocks): OK(unit 5)<br />
Controller: c1<br />
-------------<br />
Driver: 1.50.01.002<br />
Model: 7500-8<br />
FW: FE7X 1.05.00.068<br />
BIOS: BE7X 1.08.00.048<br />
Monitor: ME7X 1.01.00.040<br />
Serial #: F11605A3180167<br />
PCB: Rev3<br />
PCHIP: 1.30-33<br />
ACHIP: 3.20<br />
<br />
<br />
# of units: 2<br />
Unit 0: RAID 5 698.65 GB ( 1465185024 blocks): OK<br />
Unit 4: RAID 5 698.65 GB ( 1465185024 blocks): OK<br />
<br />
# of ports: 8<br />
Port 0: WDC WD2500JB-00GVA0 WD-WMAL71301258 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 1: WDC WD2500JB-00GVA0 WD-WMAL71322705 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 2: WDC WD2500JB-00GVA0 WD-WMAL71945050 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 3: WDC WD2500JB-00GVA0 WD-WMAL71316201 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 4: WDC WD2500JB-00GVC0 WD-WCAL78323749 232.88 GB (488397168 blocks): OK(unit 4)<br />
Port 5: WDC WD3200AAJB-00J3A0 WD-WCAV2V689068 298.09 GB (625142448 blocks): OK(unit 4)<br />
Port 6: WDC WD2500JB-00GVC0 WD-WCAL78234420 232.88 GB (488397168 blocks): OK(unit 4)<br />
Port 7: WDC WD2500JB-00GVC0 WD-WCAL78592213 232.88 GB (488397168 blocks): OK(unit 4)<br />
backup2 /d2#<br />
</pre><br />
<br />
On backup2 look for all ok, no verify.<br />
<br />
<pre><br />
[root@backup3 ~]# sh /root/verify.sh<br />
/c2 Driver Version = 1.26.02.002<br />
/c2 Model = 8006-2LP<br />
/c2 Available Memory = 512KB<br />
/c2 Firmware Version = FE8S 1.05.00.068<br />
/c2 Bios Version = BE7X 1.08.00.048<br />
/c2 Boot Loader Version = ME7X 1.01.00.040<br />
/c2 Serial Number = L018501C6481395<br />
/c2 PCB Version = Rev5<br />
/c2 PCHIP Version = 1.30-66<br />
/c2 ACHIP Version = 3.20<br />
/c2 Total Optimal Units = 1<br />
/c2 Not Optimal Units = 0<br />
<br />
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u1 RAID-1 OK - - - 931.512 ON -<br />
<br />
Port Status Unit Size Blocks Serial<br />
---------------------------------------------------------------<br />
p0 OK u1 931.51 GB 1953525168 WD-WMAW31148820<br />
p1 OK u1 931.51 GB 1953525168 WD-WCATR0277515<br />
<br />
<br />
Ctl Date Severity Alarm Message<br />
------------------------------------------------------------------------------<br />
<br />
Sending start verify message to /c2/u1 ... Done.<br />
<br />
<br />
when done run:<br />
tw_cli /c2 show alarms<br />
<br />
[root@backup3 ~]#<br />
</pre><br />
<br />
Automatically starts the verify, just run <tt>tw_cli /c2 show alarms</tt> as instructed to see the results of the verify.<br />
<br />
<pre><br />
[root@backup1 /data/deprecated]# sh /root/verify.sh<br />
/c0 Driver Version = 2.26.02.010<br />
/c0 Model = 9650SE-8LPML<br />
/c0 Available Memory = 224MB<br />
/c0 Firmware Version = FE9X 4.06.00.004<br />
/c0 Bios Version = BE9X 4.05.00.015<br />
/c0 Boot Loader Version = BL9X 3.08.00.001<br />
/c0 Serial Number = L326025A8270177<br />
/c0 PCB Version = Rev 032<br />
/c0 PCHIP Version = 2.00<br />
/c0 ACHIP Version = 1.90<br />
/c0 Number of Ports = 8<br />
/c0 Number of Drives = 6<br />
/c0 Number of Units = 1<br />
/c0 Total Optimal Units = 1<br />
/c0 Not Optimal Units = 0<br />
/c0 JBOD Export Policy = off<br />
/c0 Disk Spinup Policy = 1<br />
/c0 Spinup Stagger Time Policy (sec) = 1<br />
/c0 Auto-Carving Policy = off<br />
/c0 Auto-Carving Size = 2048 GB<br />
/c0 Auto-Rebuild Policy = on<br />
/c0 Controller Bus Type = PCIe<br />
/c0 Controller Bus Width = 1 lane<br />
/c0 Controller Bus Speed = 2.5 Gbps/lane<br />
<br />
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u0 RAID-5 OK - - 64K 4656.56 ON ON<br />
<br />
Port Status Unit Size Blocks Serial<br />
---------------------------------------------------------------<br />
p0 OK u0 931.51 GB 1953525168 9QJ1Y017<br />
p1 DEVICE-ERROR u0 931.51 GB 1953525168 9QJ1ZN07<br />
p2 OK u0 931.51 GB 1953525168 9QJ2XK1R<br />
p3 OK u0 931.51 GB 1953525168 9QJ2010B<br />
p4 OK u0 1.36 TB 2930277168 6XW0L36T<br />
p5 OK u0 931.51 GB 1953525168 WD-WMATV2444836<br />
p6 NOT-PRESENT - - - -<br />
p7 NOT-PRESENT - - - -<br />
<br />
<br />
Ctl Date Severity Alarm Message<br />
------------------------------------------------------------------------------<br />
c0 [Sat May 12 11:27:15 2012] WARNING Sector repair completed: port=0, LBA=0x6AE571C<br />
c0 [Sat May 12 19:16:21 2012] WARNING Sector repair completed: port=1, LBA=0x40E62A23<br />
c0 [Sat May 12 21:40:56 2012] INFO Verify completed: unit=0<br />
c0 [Mon May 14 00:53:53 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFA7<br />
c0 [Mon May 14 00:58:21 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAA<br />
c0 [Mon May 14 04:35:13 2012] WARNING Sector repair completed: port=0, LBA=0x8FEF2CF<br />
c0 [Mon May 14 04:38:22 2012] WARNING Sector repair completed: port=0, LBA=0x8FEF2D1<br />
c0 [Tue May 15 22:53:46 2012] WARNING Sector repair completed: port=0, LBA=0x13C2622<br />
c0 [Wed May 16 00:39:31 2012] WARNING Sector repair completed: port=0, LBA=0x365A67F<br />
c0 [Wed May 16 00:39:37 2012] WARNING Sector repair completed: port=0, LBA=0x365A685<br />
c0 [Wed May 16 00:47:18 2012] WARNING Sector repair completed: port=0, LBA=0x365A687<br />
c0 [Sat May 19 00:01:44 2012] INFO Verify started: unit=0<br />
c0 [Sat May 19 04:46:20 2012] WARNING Sector repair completed: port=0, LBA=0x365A68E<br />
c0 [Sat May 19 13:37:06 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAC<br />
c0 [Sat May 19 13:37:28 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAE<br />
c0 [Sat May 19 13:37:47 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFB1<br />
c0 [Sat May 19 13:38:00 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFB3<br />
c0 [Sat May 19 21:47:45 2012] INFO Verify completed: unit=0<br />
c0 [Wed May 23 12:21:41 2012] INFO Cache synchronization completed: unit=0<br />
c0 [Fri May 25 00:08:19 2012] WARNING Sector repair completed: port=0, LBA=0x12DA76C<br />
c0 [Fri May 25 00:08:34 2012] WARNING Sector repair completed: port=0, LBA=0x12E4901<br />
c0 [Fri May 25 00:09:33 2012] WARNING Sector repair completed: port=0, LBA=0x12DA773<br />
c0 [Fri May 25 00:39:12 2012] WARNING Sector repair completed: port=0, LBA=0x42C597B<br />
c0 [Sat May 26 00:01:45 2012] INFO Verify started: unit=0<br />
c0 [Sat May 26 00:42:05 2012] WARNING Sector repair completed: port=1, LBA=0x323C1AC<br />
c0 [Sat May 26 00:51:43 2012] WARNING Sector repair completed: port=1, LBA=0x323C1AE<br />
c0 [Sat May 26 01:54:21 2012] WARNING Sector repair completed: port=1, LBA=0x2F0D302<br />
c0 [Sat May 26 02:06:38 2012] WARNING Sector repair completed: port=0, LBA=0x12DA777<br />
c0 [Sat May 26 02:07:21 2012] WARNING Sector repair completed: port=0, LBA=0x12E48FE<br />
c0 [Sat May 26 04:20:00 2012] WARNING Sector repair completed: port=1, LBA=0x2F0D306<br />
c0 [Sat May 26 04:32:58 2012] WARNING Sector repair completed: port=1, LBA=0x323C1B1<br />
c0 [Sat May 26 04:33:21 2012] WARNING Sector repair completed: port=1, LBA=0x323C1B3<br />
c0 [Sat May 26 04:33:44 2012] WARNING Sector repair completed: port=1, LBA=0x323C1BA<br />
c0 [Sat May 26 05:24:07 2012] WARNING Sector repair completed: port=1, LBA=0x3F83862<br />
c0 [Sat May 26 05:25:09 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 06:08:13 2012] WARNING Sector repair completed: port=0, LBA=0x4CDC6A2<br />
c0 [Sat May 26 09:49:35 2012] WARNING Sector repair completed: port=1, LBA=0x6CACD4A<br />
c0 [Sat May 26 18:10:44 2012] WARNING Sector repair completed: port=1, LBA=0x18F425EA<br />
c0 [Sat May 26 19:45:40 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:22:52 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:15 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:22 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:35 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:41 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:49 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:57 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:24:02 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:54:41 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 22:00:30 2012] INFO Verify completed: unit=0<br />
c0 [Sat Jun 2 00:01:43 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 2 00:30:17 2012] WARNING Sector repair completed: port=0, LBA=0x2B911E4<br />
c0 [Sat Jun 2 00:50:57 2012] WARNING Sector repair completed: port=0, LBA=0x5A807CA6<br />
c0 [Sat Jun 2 04:13:13 2012] WARNING Sector repair completed: port=0, LBA=0x2D18291<br />
c0 [Sat Jun 2 04:13:35 2012] WARNING Sector repair completed: port=0, LBA=0x2D1829F<br />
c0 [Sat Jun 2 21:48:02 2012] INFO Verify completed: unit=0<br />
c0 [Mon Jun 4 04:40:34 2012] WARNING Sector repair completed: port=1, LBA=0x4AF8098F<br />
c0 [Tue Jun 5 00:28:19 2012] WARNING Sector repair completed: port=1, LBA=0x263C5CD<br />
c0 [Tue Jun 5 00:33:06 2012] WARNING Sector repair completed: port=1, LBA=0x263C5CF<br />
c0 [Thu Jun 7 00:34:27 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B5F<br />
c0 [Thu Jun 7 00:38:50 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B61<br />
c0 [Fri Jun 8 00:07:13 2012] WARNING Sector repair completed: port=0, LBA=0xC131F6B<br />
c0 [Sat Jun 9 00:01:41 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 9 00:29:11 2012] WARNING Sector repair completed: port=0, LBA=0x6C7614D<br />
c0 [Sat Jun 9 00:38:25 2012] WARNING Sector repair completed: port=0, LBA=0x6C76152<br />
c0 [Sat Jun 9 04:02:30 2012] WARNING Sector repair completed: port=1, LBA=0x263C5D1<br />
c0 [Sat Jun 9 04:02:52 2012] WARNING Sector repair completed: port=1, LBA=0x263C5D3<br />
c0 [Sat Jun 9 04:07:32 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E12<br />
c0 [Sat Jun 9 04:07:57 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E15<br />
c0 [Sat Jun 9 04:08:16 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E17<br />
c0 [Sat Jun 9 04:08:45 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E19<br />
c0 [Sat Jun 9 04:15:04 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B64<br />
c0 [Sat Jun 9 04:15:26 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B66<br />
c0 [Sat Jun 9 04:15:45 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B68<br />
c0 [Sat Jun 9 04:15:59 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B6C<br />
c0 [Sat Jun 9 04:16:13 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B6E<br />
c0 [Sat Jun 9 21:48:52 2012] INFO Verify completed: unit=0<br />
c0 [Thu Jun 14 00:40:10 2012] WARNING Sector repair completed: port=0, LBA=0x334F14B<br />
c0 [Sat Jun 16 00:01:38 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 16 21:16:19 2012] INFO Verify completed: unit=0<br />
c0 [Tue Jun 19 02:03:43 2012] WARNING Sector repair completed: port=1, LBA=0xFE41EAD<br />
c0 [Wed Jun 20 02:30:02 2012] WARNING Sector repair completed: port=1, LBA=0xD99145C<br />
c0 [Sat Jun 23 00:01:36 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 23 04:27:04 2012] WARNING Sector repair completed: port=1, LBA=0x2FAD311<br />
c0 [Sat Jun 23 06:52:38 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC8D<br />
c0 [Sat Jun 23 06:53:03 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC91<br />
c0 [Sat Jun 23 06:53:21 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC94<br />
c0 [Sat Jun 23 17:00:22 2012] WARNING Sector repair completed: port=1, LBA=0xF9AC7C9<br />
c0 [Sat Jun 23 21:15:19 2012] INFO Verify completed: unit=0<br />
c0 [Sat Jun 30 00:01:34 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 30 05:24:13 2012] WARNING Sector repair completed: port=0, LBA=0x3FAA9E7<br />
c0 [Sat Jun 30 14:49:39 2012] WARNING Sector repair completed: port=1, LBA=0x869931C<br />
c0 [Sat Jun 30 21:31:05 2012] INFO Verify completed: unit=0<br />
c0 [Tue Jul 3 03:40:25 2012] WARNING Sector repair completed: port=1, LBA=0xD36C7F7<br />
c0 [Fri Jul 6 02:50:18 2012] WARNING Sector repair completed: port=1, LBA=0x3562470<br />
c0 [Fri Jul 6 22:18:26 2012] WARNING Sector repair completed: port=1, LBA=0x3563173<br />
c0 [Sat Jul 7 00:01:31 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 7 00:50:16 2012] WARNING Sector repair completed: port=0, LBA=0x76EE88<br />
c0 [Sat Jul 7 00:50:39 2012] WARNING Sector repair completed: port=0, LBA=0x76EE8F<br />
c0 [Sat Jul 7 21:39:36 2012] INFO Verify completed: unit=0<br />
c0 [Sun Jul 8 02:51:05 2012] WARNING Sector repair completed: port=0, LBA=0x67759D<br />
c0 [Sun Jul 8 02:53:55 2012] WARNING Sector repair completed: port=0, LBA=0x67759B<br />
c0 [Tue Jul 10 16:17:21 2012] WARNING Sector repair completed: port=0, LBA=0x15C8C695<br />
c0 [Wed Jul 11 22:51:22 2012] WARNING Sector repair completed: port=1, LBA=0x355BBD0<br />
c0 [Sat Jul 14 00:01:28 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 14 01:33:40 2012] WARNING Sector repair completed: port=1, LBA=0x1333BCF4<br />
c0 [Sat Jul 14 03:36:23 2012] WARNING Sector repair completed: port=1, LBA=0x2174773<br />
c0 [Sat Jul 14 11:26:44 2012] WARNING Sector repair completed: port=0, LBA=0x7429AB7<br />
c0 [Sat Jul 14 16:53:50 2012] WARNING Sector repair completed: port=1, LBA=0xA17EB3F<br />
c0 [Sat Jul 14 21:19:25 2012] INFO Verify completed: unit=0<br />
c0 [Wed Jul 18 05:08:47 2012] WARNING Sector repair completed: port=1, LBA=0x17D62EDC<br />
c0 [Wed Jul 18 05:14:15 2012] WARNING Sector repair completed: port=1, LBA=0x17D62EE1<br />
c0 [Thu Jul 19 03:24:59 2012] WARNING Sector repair completed: port=0, LBA=0x7733C3D<br />
c0 [Thu Jul 19 03:25:20 2012] WARNING Sector repair completed: port=0, LBA=0x773CEA5<br />
c0 [Thu Jul 19 03:28:16 2012] WARNING Sector repair completed: port=0, LBA=0x7733C42<br />
c0 [Thu Jul 19 03:28:41 2012] WARNING Sector repair completed: port=0, LBA=0x773CEAF<br />
c0 [Sat Jul 21 00:01:26 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 21 03:07:30 2012] WARNING Sector repair completed: port=1, LBA=0x1CC6936<br />
c0 [Sat Jul 21 03:07:52 2012] WARNING Sector repair completed: port=1, LBA=0x1CC6938<br />
c0 [Sat Jul 21 03:08:11 2012] WARNING Sector repair completed: port=1, LBA=0x1CC693A<br />
c0 [Sat Jul 21 16:43:56 2012] WARNING Sector repair completed: port=0, LBA=0xD04C914<br />
c0 [Sat Jul 21 16:45:31 2012] WARNING Sector repair completed: port=1, LBA=0xD456973<br />
c0 [Sat Jul 21 21:14:29 2012] INFO Verify completed: unit=0<br />
c0 [Wed Jul 25 03:37:25 2012] WARNING Sector repair completed: port=0, LBA=0x1F8E6C43<br />
c0 [Sat Jul 28 00:01:24 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 28 01:45:27 2012] WARNING Sector repair completed: port=0, LBA=0x11584AD<br />
c0 [Sat Jul 28 18:54:25 2012] WARNING Sector repair completed: port=1, LBA=0x447C3E6C<br />
c0 [Sat Jul 28 21:13:46 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 1 03:20:11 2012] WARNING Sector repair completed: port=0, LBA=0x805FEF<br />
c0 [Fri Aug 3 00:50:03 2012] WARNING Sector repair completed: port=0, LBA=0xCED0ACA<br />
c0 [Sat Aug 4 00:01:22 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 4 00:52:51 2012] WARNING Sector repair completed: port=0, LBA=0x805FF3<br />
c0 [Sat Aug 4 00:53:14 2012] WARNING Sector repair completed: port=0, LBA=0x805FF5<br />
c0 [Sat Aug 4 00:53:33 2012] WARNING Sector repair completed: port=0, LBA=0x805FF7<br />
c0 [Sat Aug 4 00:53:47 2012] WARNING Sector repair completed: port=0, LBA=0x805FF9<br />
c0 [Sat Aug 4 00:54:00 2012] WARNING Sector repair completed: port=0, LBA=0x805FFB<br />
c0 [Sat Aug 4 00:54:14 2012] WARNING Sector repair completed: port=0, LBA=0x805FFD<br />
c0 [Sat Aug 4 00:54:27 2012] WARNING Sector repair completed: port=0, LBA=0x805FFF<br />
c0 [Sat Aug 4 04:43:12 2012] WARNING Sector repair completed: port=1, LBA=0x16974289<br />
c0 [Sat Aug 4 04:58:17 2012] WARNING Sector repair completed: port=1, LBA=0x1697428E<br />
c0 [Sat Aug 4 20:54:53 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 8 03:21:55 2012] ERROR Drive timeout detected: port=1<br />
c0 [Wed Aug 8 15:31:44 2012] WARNING Sector repair completed: port=0, LBA=0x1A366CD3<br />
c0 [Sat Aug 11 00:01:21 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 11 20:40:51 2012] INFO Verify completed: unit=0<br />
c0 [Thu Aug 16 05:10:55 2012] WARNING Sector repair completed: port=0, LBA=0x1C22593<br />
c0 [Sat Aug 18 00:01:18 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 18 03:00:20 2012] WARNING Sector repair completed: port=0, LBA=0x1C225A5<br />
c0 [Sat Aug 18 03:43:00 2012] WARNING Sector repair completed: port=1, LBA=0x23EE91E<br />
c0 [Sat Aug 18 03:43:23 2012] WARNING Sector repair completed: port=1, LBA=0x23EE920<br />
c0 [Sat Aug 18 17:00:06 2012] WARNING Sector repair completed: port=1, LBA=0x137D066A<br />
c0 [Sat Aug 18 17:00:29 2012] WARNING Sector repair completed: port=1, LBA=0x137D066D<br />
c0 [Sat Aug 18 21:13:01 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 22 01:36:08 2012] WARNING Sector repair completed: port=0, LBA=0x2560A0F<br />
c0 [Wed Aug 22 01:37:42 2012] WARNING Sector repair completed: port=0, LBA=0x2560A13<br />
c0 [Fri Aug 24 04:01:36 2012] WARNING Sector repair completed: port=1, LBA=0x55C1A5DF<br />
c0 [Fri Aug 24 05:02:06 2012] WARNING Sector repair completed: port=1, LBA=0xCE3378A<br />
c0 [Sat Aug 25 00:01:17 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 25 00:31:06 2012] WARNING Sector repair completed: port=1, LBA=0x50F65D<br />
c0 [Sat Aug 25 00:39:52 2012] WARNING Sector repair completed: port=0, LBA=0x678FF4<br />
c0 [Sat Aug 25 03:43:15 2012] WARNING Sector repair completed: port=0, LBA=0x2560A15<br />
c0 [Sat Aug 25 03:43:39 2012] WARNING Sector repair completed: port=0, LBA=0x2560A19<br />
c0 [Sat Aug 25 03:43:58 2012] WARNING Sector repair completed: port=0, LBA=0x2560A1B<br />
c0 [Sat Aug 25 03:44:30 2012] WARNING Sector repair completed: port=0, LBA=0x2560A21<br />
c0 [Sat Aug 25 20:58:14 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 29 04:57:15 2012] WARNING Sector repair completed: port=1, LBA=0xF3957EB<br />
c0 [Sat Sep 1 00:01:15 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 1 03:21:52 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC86<br />
c0 [Sat Sep 1 03:22:15 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC88<br />
c0 [Sat Sep 1 03:22:34 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC8A<br />
c0 [Sat Sep 1 03:22:47 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC8C<br />
c0 [Sat Sep 1 17:17:22 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD1<br />
c0 [Sat Sep 1 17:17:45 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD3<br />
c0 [Sat Sep 1 17:18:04 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD5<br />
c0 [Sat Sep 1 21:36:56 2012] INFO Verify completed: unit=0<br />
c0 [Thu Sep 6 00:07:30 2012] WARNING Sector repair completed: port=0, LBA=0xDA3C64B<br />
c0 [Thu Sep 6 00:32:56 2012] WARNING Sector repair completed: port=1, LBA=0x6BBA816<br />
c0 [Sat Sep 8 00:01:13 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 8 00:09:56 2012] WARNING Sector repair completed: port=0, LBA=0xDEBC958<br />
c0 [Sat Sep 8 04:38:45 2012] WARNING Sector repair completed: port=0, LBA=0x38D254F<br />
c0 [Sat Sep 8 20:44:50 2012] INFO Verify completed: unit=0<br />
c0 [Mon Sep 10 01:26:34 2012] WARNING Sector repair completed: port=1, LBA=0xFFD8D5E<br />
c0 [Wed Sep 12 00:33:48 2012] WARNING Sector repair completed: port=1, LBA=0xE8DB928<br />
c0 [Wed Sep 12 00:36:33 2012] WARNING Sector repair completed: port=1, LBA=0x6D49411<br />
c0 [Fri Sep 14 01:59:39 2012] WARNING Sector repair completed: port=0, LBA=0x1467F1C<br />
c0 [Fri Sep 14 02:08:27 2012] WARNING Sector repair completed: port=1, LBA=0x14C8ABD<br />
c0 [Fri Sep 14 03:54:47 2012] WARNING Sector repair completed: port=0, LBA=0x1580C915<br />
c0 [Sat Sep 15 00:01:11 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 15 02:38:14 2012] WARNING Sector repair completed: port=0, LBA=0x1C178973<br />
c0 [Sat Sep 15 02:59:02 2012] WARNING Sector repair completed: port=0, LBA=0x1C178975<br />
c0 [Sat Sep 15 04:47:08 2012] WARNING Sector repair completed: port=0, LBA=0x3FA0356<br />
c0 [Sat Sep 15 04:47:31 2012] WARNING Sector repair completed: port=0, LBA=0x3FA0359<br />
c0 [Sat Sep 15 10:41:59 2012] WARNING Sector repair completed: port=0, LBA=0x6DFD1EC<br />
c0 [Sat Sep 15 13:25:23 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD100<br />
c0 [Sat Sep 15 13:25:31 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD104<br />
c0 [Sat Sep 15 13:25:54 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD106<br />
c0 [Sat Sep 15 17:10:50 2012] WARNING Sector repair completed: port=0, LBA=0x1C178977<br />
c0 [Sat Sep 15 20:59:57 2012] INFO Verify completed: unit=0<br />
c0 [Tue Sep 18 01:17:18 2012] WARNING Sector repair completed: port=1, LBA=0x803B05B<br />
c0 [Sat Sep 22 00:01:10 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 22 20:54:31 2012] INFO Verify completed: unit=0<br />
c0 [Tue Sep 25 01:56:47 2012] WARNING Sector repair completed: port=0, LBA=0x26E3909<br />
c0 [Sat Sep 29 00:01:08 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 29 02:04:14 2012] WARNING Sector repair completed: port=0, LBA=0x146AC03<br />
c0 [Sat Sep 29 10:58:39 2012] WARNING Sector repair completed: port=0, LBA=0x6D4EB0E<br />
c0 [Sat Sep 29 10:59:02 2012] WARNING Sector repair completed: port=0, LBA=0x6D4EB14<br />
c0 [Sat Sep 29 11:22:44 2012] WARNING Sector repair completed: port=0, LBA=0x6F79623<br />
c0 [Sat Sep 29 13:50:48 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D65E<br />
c0 [Sat Sep 29 13:51:11 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D661<br />
c0 [Sat Sep 29 13:51:30 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D663<br />
c0 [Sat Sep 29 20:57:34 2012] INFO Verify completed: unit=0<br />
c0 [Mon Oct 1 04:47:24 2012] WARNING Sector repair completed: port=0, LBA=0xC5BC6F2<br />
c0 [Tue Oct 2 02:00:27 2012] WARNING Sector repair completed: port=0, LBA=0x1547667<br />
c0 [Tue Oct 2 02:01:56 2012] WARNING Sector repair completed: port=0, LBA=0x154766F<br />
c0 [Tue Oct 2 05:02:31 2012] WARNING Sector repair completed: port=1, LBA=0xD67D054<br />
c0 [Tue Oct 2 05:04:14 2012] WARNING Sector repair completed: port=1, LBA=0xD67D056<br />
c0 [Wed Oct 3 01:22:12 2012] WARNING Sector repair completed: port=1, LBA=0x12AAF8CA<br />
c0 [Thu Oct 4 04:29:22 2012] WARNING Sector repair completed: port=0, LBA=0x13E6F992<br />
c0 [Thu Oct 4 05:10:51 2012] WARNING Sector repair completed: port=0, LBA=0x1C252A4<br />
c0 [Sat Oct 6 00:01:07 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 6 19:41:18 2012] WARNING Sector repair completed: port=1, LBA=0x5A5C3AE8<br />
c0 [Sat Oct 6 21:01:05 2012] INFO Verify completed: unit=0<br />
c0 [Mon Oct 8 00:32:06 2012] WARNING Sector repair completed: port=0, LBA=0x6C60D3E<br />
c0 [Tue Oct 9 03:51:03 2012] WARNING Sector repair completed: port=1, LBA=0x89B5EC9<br />
c0 [Thu Oct 11 04:21:17 2012] WARNING Sector repair completed: port=1, LBA=0x13F85833<br />
c0 [Sat Oct 13 00:01:05 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 13 05:12:40 2012] WARNING Sector repair completed: port=0, LBA=0x3FA5134<br />
c0 [Sat Oct 13 21:08:35 2012] INFO Verify completed: unit=0<br />
c0 [Tue Oct 16 03:53:50 2012] WARNING Sector repair completed: port=1, LBA=0x148AA1BD<br />
c0 [Thu Oct 18 03:20:30 2012] WARNING Sector repair completed: port=0, LBA=0x1C8DABCB<br />
c0 [Thu Oct 18 04:52:50 2012] WARNING Sector repair completed: port=0, LBA=0xE879057<br />
c0 [Sat Oct 20 00:01:04 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 20 02:19:25 2012] WARNING Sector repair completed: port=1, LBA=0x174B012<br />
c0 [Sat Oct 20 03:41:38 2012] WARNING Sector repair completed: port=0, LBA=0x256D93B<br />
c0 [Sat Oct 20 03:42:01 2012] WARNING Sector repair completed: port=0, LBA=0x256D93D<br />
c0 [Sat Oct 20 03:42:40 2012] WARNING Sector repair completed: port=0, LBA=0x256D940<br />
c0 [Sat Oct 20 03:42:59 2012] WARNING Sector repair completed: port=0, LBA=0x256D942<br />
c0 [Sat Oct 20 03:43:12 2012] WARNING Sector repair completed: port=0, LBA=0x256D944<br />
c0 [Sat Oct 20 03:43:26 2012] WARNING Sector repair completed: port=0, LBA=0x256D948<br />
c0 [Sat Oct 20 16:37:52 2012] WARNING Sector repair completed: port=0, LBA=0xE879060<br />
c0 [Sat Oct 20 16:38:15 2012] WARNING Sector repair completed: port=0, LBA=0xE879062<br />
c0 [Sat Oct 20 21:00:18 2012] INFO Verify completed: unit=0<br />
c0 [Sat Oct 20 23:49:01 2012] WARNING Sector repair completed: port=1, LBA=0x4473E908<br />
c0 [Sun Oct 21 03:42:26 2012] WARNING Sector repair completed: port=1, LBA=0x175BADD5<br />
c0 [Tue Oct 23 01:09:04 2012] WARNING Sector repair completed: port=1, LBA=0x6E524860<br />
c0 [Fri Oct 26 03:21:25 2012] WARNING Sector repair completed: port=0, LBA=0x802C61<br />
c0 [Fri Oct 26 04:22:21 2012] WARNING Sector repair completed: port=0, LBA=0x176353CD<br />
c0 [Sat Oct 27 00:01:03 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 27 00:49:35 2012] WARNING Sector repair completed: port=0, LBA=0x802C65<br />
c0 [Sat Oct 27 17:02:24 2012] WARNING Sector repair completed: port=1, LBA=0xC1FF26D<br />
c0 [Sat Oct 27 17:09:06 2012] WARNING Sector repair completed: port=0, LBA=0xDF621AD<br />
c0 [Sat Oct 27 21:30:57 2012] INFO Verify completed: unit=0<br />
c0 [Tue Oct 30 00:20:46 2012] WARNING Sector repair completed: port=0, LBA=0xE9FE2AB<br />
c0 [Wed Oct 31 02:02:03 2012] WARNING Sector repair completed: port=0, LBA=0x1460C25<br />
c0 [Wed Oct 31 02:04:05 2012] WARNING Sector repair completed: port=0, LBA=0x1460C28<br />
c0 [Thu Nov 1 00:48:34 2012] WARNING Sector repair completed: port=1, LBA=0xA7C92BE<br />
c0 [Thu Nov 1 05:04:45 2012] WARNING Sector repair completed: port=0, LBA=0x1C252C2<br />
<br />
[root@backup1 /data/deprecated]#<br />
</pre><br />
<br />
Look for failed drives and errors. Obviously from the above we need to probably replace drives 0 and 1 and drive 1 is even showing as having problems, yet the RAID array is healthy, amazingly. You also see the automatic verifies.<br />
<br />
Note: when rebuilding a degraded mirror, you will see no progress as it rebuilds in the cli<br />
<pre>Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u0 RAID-5 REBUILDING 0 - 64K 4656.56 OFF ON</pre><br />
<br />
<br />
=== areca ===<br />
<br />
We are using an areca controller on backup3. <br />
<br />
<pre><br />
[root@newbackup3 ~]# sh /root/verify.sh<br />
# Name Raid Name Level Capacity Ch/Id/Lun State<br />
===============================================================================<br />
1 ARC-1160-VOL#00 Raid Set # 00 Raid5 5000.0GB 00/00/00 Checking(19.7%)<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
# Name Disks TotalCap FreeCap DiskChannels State<br />
===============================================================================<br />
1 Raid Set # 00 6 6000.0GB 0.0GB 123456 Checking<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
Date-Time Device Event Type Elapsed Time Errors<br />
===============================================================================<br />
2012-12-05 20:40:58 ARC-1160-VOL#00 Start Checking<br />
2012-12-01 05:06:04 ARC-1160-VOL#00 Complete Init 027:30:45<br />
2012-11-30 01:35:19 ARC-1160-VOL#00 Start Initialize<br />
2026-08-06 01:34:52 H/W MONITOR Raid Powered On<br />
2012-11-30 01:33:36 ARC-1160-VOL#00 Stop Initialization 000:31:48<br />
2012-11-30 01:01:47 ARC-1160-VOL#00 Start Initialize<br />
2026-08-06 00:58:13 H/W MONITOR Raid Powered On<br />
2012-11-30 00:57:26 ARC-1160-VOL#00 Stop Initialization 000:57:07<br />
2012-11-30 00:00:19 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 23:56:48 H/W MONITOR Raid Powered On<br />
2026-08-05 23:52:58 H/W MONITOR Raid Powered On<br />
2026-08-05 23:50:14 H/W MONITOR Raid Powered On<br />
2026-08-05 23:43:30 H/W MONITOR Raid Powered On<br />
2012-11-29 23:10:07 ARC-1160-VOL#00 Stop Initialization 000:00:56<br />
2012-11-29 23:09:11 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 23:08:57 H/W MONITOR Raid Powered On<br />
2012-11-29 23:08:10 ARC-1160-VOL#00 Stop Initialization 000:20:41<br />
2012-11-29 22:47:29 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 22:46:59 H/W MONITOR Raid Powered On<br />
2026-08-05 22:45:55 H/W MONITOR Raid Powered On<br />
2026-08-05 22:44:53 H/W MONITOR Raid Powered On<br />
2026-08-05 22:42:06 H/W MONITOR Raid Powered On<br />
2026-08-05 22:40:50 H/W MONITOR Raid Powered On<br />
2012-11-29 22:40:04 ARC-1160-VOL#00 Stop Initialization 000:24:25<br />
2012-11-29 22:15:38 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 22:15:11 000:000001215B00 Restart Init LBA Point<br />
2026-08-05 22:15:10 H/W MONITOR Raid Powered On<br />
2012-11-29 21:56:38 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 21:56:12 H/W MONITOR Raid Powered On<br />
2026-08-05 21:56:04 IDE Channel #03 Device Inserted<br />
2012-11-29 21:55:13 IDE Channel #04 Device Inserted<br />
2012-11-29 21:55:03 IDE Channel #02 Device Inserted<br />
2026-08-05 21:53:09 H/W MONITOR Raid Powered On<br />
2026-08-05 20:51:46 H/W MONITOR Raid Powered On<br />
2026-08-05 20:49:56 H/W MONITOR Raid Powered On<br />
2026-08-05 20:48:29 H/W MONITOR Raid Powered On<br />
2026-08-05 20:46:29 H/W MONITOR Raid Powered On<br />
2026-08-05 20:44:49 H/W MONITOR Raid Powered On<br />
2026-08-05 20:43:01 H/W MONITOR Raid Powered On<br />
2026-08-05 20:36:25 H/W MONITOR Raid Powered On<br />
2026-08-05 20:31:18 H/W MONITOR Raid Powered On<br />
2026-08-05 20:30:08 H/W MONITOR Raid Powered On<br />
2026-08-05 20:08:40 H/W MONITOR Raid Powered On<br />
2026-08-05 20:06:11 H/W MONITOR Raid Powered On<br />
2026-08-05 20:05:14 H/W MONITOR Raid Powered On<br />
2026-08-05 20:03:58 H/W MONITOR Raid Powered On<br />
2026-08-05 20:00:56 H/W MONITOR Raid Powered On<br />
2026-08-05 19:57:57 H/W MONITOR Raid Powered On<br />
2026-08-05 19:56:15 H/W MONITOR Raid Powered On<br />
2026-08-05 19:55:05 H/W MONITOR Raid Powered On<br />
2026-08-05 17:24:36 H/W MONITOR Raid Powered On<br />
2026-08-05 17:22:43 H/W MONITOR Raid Powered On<br />
2026-08-05 04:50:42 H/W MONITOR Raid Powered On<br />
2026-08-05 04:47:33 H/W MONITOR Raid Powered On<br />
2026-08-05 04:43:57 H/W MONITOR Raid Powered On<br />
2026-08-05 04:18:52 H/W MONITOR Raid Powered On<br />
2026-08-05 04:17:30 H/W MONITOR Raid Powered On<br />
2026-08-05 04:13:30 H/W MONITOR Raid Powered On<br />
2026-08-05 04:10:26 H/W MONITOR Raid Powered On<br />
2026-08-05 04:09:23 H/W MONITOR Raid Powered On<br />
2026-08-05 00:08:09 H/W MONITOR Raid Powered On<br />
2026-08-05 00:07:12 H/W MONITOR Raid Powered On<br />
2026-08-05 00:05:51 H/W MONITOR Raid Powered On<br />
2026-08-05 00:04:27 H/W MONITOR Raid Powered On<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
press enter when ready to run verify<br />
</pre><br />
<br />
Look for failed drives and errors. <br />
<br />
When it proceed's to verifying, you can confirm with:<br />
<br />
<pre><br />
[root@newbackup3 ~]# cli64 vsf info<br />
# Name Raid Name Level Capacity Ch/Id/Lun State<br />
===============================================================================<br />
1 ARC-1160-VOL#00 Raid Set # 00 Raid5 5000.0GB 00/00/00 Checking(22.5%)<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
[root@newbackup3 ~]#<br />
</pre><br />
<br />
== Update OS list ==<br />
<br />
# check for any new VZ templates we want to offer: <tt>vzup2date -z</tt><br />
# see if there's any OS's we want to include in our colo install list. Update 2 places: <tt>signup/html/colo_quote.html</tt> & <tt>signup/html/step1.html</tt><br />
# update the mgmt database (ref_templates table, ref_systems table).<br />
<br />
= Infrequent tasks =<br />
<br />
== Free up space on gateway ==<br />
<br />
<pre>newgateway /var/spool# cd clientmqueue/<br />
newgateway /var/spool/clientmqueue# sh<br />
# for f in `ls`; do rm $f; done<br />
exit</pre><br />
<br />
== Free up space on mail ==<br />
<br />
You can clear out root mail:<br />
<pre><br />
mail /var/log# ll -h /var/mail/root<br />
-rw------- 1 root mail 543K Dec 19 13:05 /var/mail/root<br />
mail /var/log# rm /var/mail/root<br />
</pre><br />
<br />
Or you can archive mail logs:<br />
<pre><br />
mail /var/log# ls -l htt*<br />
-rw-r--r-- 1 root wheel 297436931 Dec 19 13:26 httpd-access.log<br />
-rw-r--r-- 1 root wheel 9824324 Jul 4 11:34 httpd-access.log.old.0.gz<br />
-rw-r--r-- 1 root wheel 6884137 Mar 17 2012 httpd-access.log.old.1.gz<br />
-rw-r--r-- 1 root wheel 18557444 Dec 3 2009 httpd-access.log.old.10.gz<br />
-rw-r--r-- 1 root wheel 14740263 Jan 9 2007 httpd-access.log.old.11.gz<br />
-rw-r--r-- 1 root wheel 14209465 Nov 28 2007 httpd-access.log.old.12.gz<br />
-rw-r--r-- 1 root wheel 16874396 Feb 19 2012 httpd-access.log.old.3.gz<br />
-rw-r--r-- 1 root wheel 14554859 Jul 22 2011 httpd-access.log.old.4.gz<br />
-rw-r--r-- 1 root wheel 10513227 Feb 18 2011 httpd-access.log.old.5.gz<br />
-rw-r--r-- 1 root wheel 7201946 Oct 29 2010 httpd-access.log.old.6.gz<br />
-rw-r--r-- 1 root wheel 10062537 May 6 2010 httpd-access.log.old.7.gz<br />
-rw-r--r-- 1 root wheel 10157042 Aug 12 2010 httpd-access.log.old.8.gz<br />
-rw-r--r-- 1 root wheel 11909534 Mar 4 2010 httpd-access.log.old.9.gz<br />
-rw-r--r-- 1 root wheel 59030930 Dec 19 13:01 httpd-error.log<br />
-rw-r--r-- 1 root wheel 3413134 Mar 4 2010 httpd-error.log.0.gz<br />
-rw-r--r-- 1 root wheel 795515 May 1 2007 httpd-error.log.1.gz<br />
-rw-r--r-- 1 root wheel 1142153 Nov 30 2007 httpd-error.log.2.gz<br />
-rw-r--r-- 1 root wheel 2325801 Feb 18 2011 httpd-error.log.gz<br />
<br />
mail /var/log# sh<br />
# for f in 12 11 10 9 8 7 6 5 4 3 2 1 0; do g=`echo $f+1|bc`; mv httpd-access.log.old.$f.gz httpd-access.log.old.$g.gz; done<br />
# mv httpd-access.log httpd-access.log.old.0<br />
# touch httpd-access.log<br />
# apachectl restart<br />
# gzip httpd-access.log.old.0<br />
<br />
# for f in 2 1 0; do g=`echo $f+1|bc`; mv httpd-error.log.$f.gz httpd-error.log.$g.gz; done<br />
# mv httpd-error.log httpd-error.log.0<br />
# touch httpd-error.log<br />
# apachectl restart<br />
# gzip httpd-error.log.0<br />
# exit<br />
</pre><br />
<br />
== Free up space on bwdb2 ==<br />
<br />
You can either remove items from <tt>/usr/home/archive</tt> or you can scp them to backup3:/data/bwdb2/archive .<br />
<br />
== Free up space on backup1 ==<br />
<br />
backup1 is our primary customer backup system. As usage grows over time, it needs to be regularly purged of old files. The easiest way to do this is by removing deprecated files. These mostly consist of cancelled customers or temporary dump/storage files (created during dump/restores). Our standard policy is to hang onto cancelled customers for 6mos after which we remove their files (as far as customers know their data is purged immediately, but we hang onto it just in case.. and in some cases we cancel a server due to non payment so this makes it easy to restore their system). To find files to remove:<br />
<br />
<pre>[root@backup1 ~]# cd /data/deprecated/<br />
[root@backup1 /data/deprecated]# ls<br />
2101-migrated-20120317.tgz old-683-cxld-20121021.tgz<br />
69.55.230.2-wwwbackup old-744-cxld-20120708.tgz<br />
991-DONTDELETE.tgz old-809-cxld-20120609.tgz<br />
archive-col02050-mdfile-cxld-20120409.gz old-854-cxld-20120621.tgz<br />
col01371.tgz old-931-cxld-20060513.tgz<br />
deleteme_ubuntu-10.10-x86_20111205 old-col00123-mdfile-noarchive-20120417.gz<br />
jail10_old old-col00147-vnfile-cxld-20120828.gz<br />
jail14_rsync_old old-col00419-dump-cxld-20120224.gz<br />
jail15_old old-col01098-vnfile-cxld-20120827.gz<br />
jail3_old old-col01278-dump-cxld-20120822<br />
jail4_old old-col01517-dump-cxld-20120828<br />
jail5_old old-col01669-dump-cxld-20120203.gz<br />
old-1009-cxld-20120608.tgz old-col01687-dump-cxld-20120909<br />
old-1012-cxld-20120411.tgz old-col01790-dump-cxld-20120828<br />
old-1052-cxld-20120721.tgz old-col01812-dump-cxld-20120820<br />
old-10631-cxld-20120622.tgz old-col01938-mdfile-cxld-20120619.gz<br />
old-10632-cxld-20120622.tgz old-col02095-mdfile-noarchive-20120523.gz<br />
old-10633-cxld-20120622.tgz olddebian-3.0-v15-20110610.tgz<br />
old-1236-cxld-20120621.tgz oldmod_frontpage-deb30-v15-20110610.tgz<br />
old-1381-cxld-20120404.tgz oldmod_perl-deb30-v15-20110610.tgz<br />
old-1422-cxld-20120721.tgz oldmod_ssl-deb30-v15-20110610.tgz<br />
old-14681-cxld-20120619.tgz oldmysql-deb30-v15-20110610.tgz<br />
old-1544-cxld-20120626.tgz oldproftpd-deb30-v15-20110610.tgz<br />
old-18351-cxld-20120605.tgz old_virt14<br />
old-1853-cxld-20120910.tgz old_virt18<br />
old-1963-cxld-20120206.tgz oldwebmin-deb30-v15-20110610.tgz<br />
old-1967-cxld-20120605.tgz suse.virt11.20120421.tgz<br />
old-1981-noarchive-20120729.tgz virt11<br />
old-2030-migrated-noarchive-20120727.tgz virt12_old<br />
old-2037-cxld-20120716.tgz virt13_old<br />
old-2065-cxld-20120727.tgz virt16_old<br />
old-2068-cxld-20120424.tgz virt4_old<br />
old-2085-cxld-20120531.tgz virt5_old<br />
old-364-cxld-20120904.tgz virt6_old<br />
old-446-cxld-20120512.tgz virt7_old<br />
old-613-cxld-20120601.tgz virt8_old<br />
[root@backup1 /data/deprecated]#<br />
</pre><br />
<br />
virtX_old and jailX_old are permanently archived, so ignore those as well as anything else marked not to delete or otherwise suspicious. Likewise, probably a good idea to try to hang onto oldTEMPLATE.gz as long as we can as well.<br />
Most of the stuff we want to delete is dated when it was deprecated, making this easy. So to remove files from 6 mos ago (running this in Oct):<br />
<br />
<pre>[root@backup1 /data/deprecated]# ls old*201204*<br />
old-1012-cxld-20120411.tgz old-2068-cxld-20120424.tgz<br />
old-1381-cxld-20120404.tgz old-col00123-mdfile-noarchive-20120417.gz<br />
[root@backup1 /data/deprecated]# rm old*201204*<br />
</pre><br />
<br />
Every few months you will also want to remove some of the snapshot archives for mail. We typically save the 1st, 10th, and 20th of each month. To do this you set aside the dates you want to save then remove months at a time, followed by restoring the set aside dates. Here's how that works:<br />
<pre>[root@backup1 /data/www/daily]# ls<br />
05 08-10-11 10-04-10 11-10-10 12-07-29 12-09-21 12-11-14<br />
06 08-10-21 10-04-20 11-10-20 12-07-30 12-09-22 12-11-15<br />
06-06-01-usr-home.tgz 08-11-01 10-05-01 11-11-01 12-07-31 12-09-23 12-11-16<br />
06-07-01-usr-home.tgz 08-11-10 10-05-11 11-11-10 12-08-01 12-09-24 12-11-17<br />
06-08-01-usr-home.tgz 08-11-20 10-05-20 11-11-20 12-08-02 12-09-25 12-11-18<br />
06-09-01-usr-home.tgz 08-12-01 10-06-01 11-12-01 12-08-03 12-09-26 12-11-19<br />
06-11-10 08-12-10 10-06-10 11-12-10 12-08-04 12-09-27 12-11-20<br />
06-12-21 08-12-20 10-06-20 11-12-20 12-08-05 12-09-28 12-11-21<br />
07-01-10 09-01-01 10-07-01 12-01-01 12-08-06 12-09-29 12-11-22<br />
07-01-20 09-01-10 10-07-10 12-01-10 12-08-07 12-09-30 12-11-23<br />
07-02-10 09-01-20 10-07-20 12-01-20 12-08-08 12-10-01 12-11-24<br />
07-02-20 09-02-01 10-08-01 12-02-01 12-08-09 12-10-02 12-11-25<br />
07-03-01 09-02-10 10-08-10 12-02-10 12-08-10 12-10-03 12-11-26<br />
07-03-20 09-02-20 10-08-20 12-02-20 12-08-11 12-10-04 12-11-27<br />
07-04-01 09-03-01 10-09-01 12-03-01 12-08-12 12-10-05 12-11-28<br />
07-04-10 09-03-10 10-09-10 12-03-10 12-08-13 12-10-06 12-11-29<br />
07-04-20 09-03-20 10-09-20 12-03-20 12-08-14 12-10-07 12-11-30<br />
07-05-01 09-04-01 10-10-01 12-04-01 12-08-15 12-10-08 12-12-01<br />
07-05-10 09-04-10 10-10-10 12-04-10 12-08-16 12-10-09 12-12-02<br />
07-05-20 09-04-20 10-10-20 12-04-20 12-08-17 12-10-10 12-12-03<br />
07-06-01 09-05-01 10-11-01 12-05-01 12-08-18 12-10-11 12-12-04<br />
07-06-10 09-05-10 10-11-10 12-05-10 12-08-19 12-10-12 12-12-05<br />
07-06-20 09-05-20 10-11-20 12-05-20 12-08-20 12-10-13 12-12-06<br />
07-07-20 09-06-01 10-12-01 12-06-01 12-08-21 12-10-14 12-12-07<br />
07-08-10 09-06-10 10-12-10 12-06-10 12-08-22 12-10-15 12-12-08<br />
07-08-20 09-06-20 10-12-20 12-06-20 12-08-23 12-10-16 12-12-09<br />
07-09-01 09-07-01 11-01-01 12-07-01 12-08-24 12-10-17 12-12-10<br />
07-10-01 09-07-10 11-01-10 12-07-02 12-08-25 12-10-18 12-12-11<br />
07-10-10 09-07-20 11-01-21 12-07-03 12-08-26 12-10-19 12-12-12<br />
07-10-20 09-08-01 11-02-01 12-07-04 12-08-27 12-10-20 12-12-13<br />
07-12-01 09-08-10 11-02-10 12-07-05 12-08-28 12-10-21 12-12-14<br />
07-12-10 09-08-20 11-02-20 12-07-06 12-08-29 12-10-22 12-12-15<br />
08-01-01 09-09-01 11-03-01 12-07-07 12-08-30 12-10-23 12-12-16<br />
08-01-20 09-09-10 11-03-10 12-07-08 12-08-31 12-10-24 12-12-17<br />
08-02-20 09-09-20 11-03-20 12-07-09 12-09-01 12-10-25 12-12-18<br />
08-03-01 09-10-01 11-04-01 12-07-10 12-09-02 12-10-26 12-12-19<br />
08-03-10 09-10-10 11-04-10 12-07-11 12-09-03 12-10-27 12-12-20<br />
08-03-20 09-10-20 11-04-20 12-07-12 12-09-04 12-10-28 12-12-21<br />
08-04-01 09-11-01 11-05-01 12-07-13 12-09-05 12-10-29 12-12-22<br />
08-04-20 09-11-10 11-05-10 12-07-14 12-09-06 12-10-30 12-12-23<br />
08-05-01 09-11-20 11-05-20 12-07-15 12-09-07 12-10-31 12-12-24<br />
08-05-10 09-12-01 11-06-01 12-07-16 12-09-08 12-11-01 12-12-25<br />
08-06-10 09-12-10 11-06-10 12-07-17 12-09-09 12-11-02 12-12-26<br />
08-06-20 09-12-20 11-06-20 12-07-18 12-09-10 12-11-03 12-12-27<br />
08-07-02 10-01-01 11-07-01 12-07-19 12-09-11 12-11-04 12-12-28<br />
08-07-10 10-01-10 11-07-10 12-07-20 12-09-12 12-11-05 2008-10-23<br />
08-07-20 10-01-20 11-07-20 12-07-21 12-09-13 12-11-06 bb.tgz<br />
08-08-01 10-02-01 11-08-01 12-07-22 12-09-14 12-11-07 boot<br />
08-08-10 10-02-10 11-08-10 12-07-23 12-09-15 12-11-08 current<br />
08-08-21 10-02-20 11-08-20 12-07-24 12-09-16 12-11-09 hold<br />
08-09-01 10-03-01 11-09-01 12-07-25 12-09-17 12-11-10<br />
08-09-10 10-03-10 11-09-10 12-07-26 12-09-18 12-11-11<br />
08-09-21 10-03-20 11-09-20 12-07-27 12-09-19 12-11-12<br />
08-10-01 10-04-01 11-10-01 12-07-28 12-09-20 12-11-13<br />
[root@backup1 /data/www/daily]#<br />
</pre><br />
<br />
So we see that everything up to July 2012 has been pruned. To prune July 2012 we do the following:<br />
<br />
<pre>mv 12-07-01 hold<br />
mv 12-07-10 hold<br />
mv 12-07-20 hold<br />
rm -fr 12-07*<br />
mv hold/* .</pre></div>75.80.128.147https://wiki.jcihosting.com/index.php?title=Routine_Maintenance&diff=2204Routine Maintenance2020-02-25T17:59:05Z<p>75.80.128.147: /* Outgoing */</p>
<hr />
<div>= Daily Tasks =<br />
<br />
== check load graphs == <br />
<br />
Click on the Load link in mgmt <br />
<br />
This screen shows you load levels on our servers and network traffic for critical machines (firewalls, backup servers).<br />
<br />
If you see load high or increasing <br />
<br />
FreeBSD: <br />
run [[VPS_Management#jtop|jtop]] (or [[VPS_Management#jt|jt]] > 7.x) and see if there are any runaway processes. Here are some examples of entries in top that are<br />
definitely runaway processes:<br />
<br />
<pre>79481 root 64 0 2256K 1056K CPU1 1 58:16 87.40% 87.40% nano<br />
50650 1000 64 0 1852K 1112K RUN 0 207.9H 84.08% 84.08% screen<br />
14829 www 2 0 39100K 31736K accept 0 104:24 46.54% 6.54% httpd<br />
42065 root 61 0 1300K 844K RUN 1 47.8H 91.36% 91.36% ee<br />
1328 www 56 0 18440K 10796K CPU1 0 64.4H 97.71% 97.71% httpd<br />
26251 user 57 0 6124K 1160K CPU1 1 82.9H 98.44% 98.44% screen<br />
89874 root 60 0 1352K 892K RUN 1 33.8H 65.82% 65.82% dialog<br />
38656 1000 64 0 3088K 2136K CPU0 0 806:13 97.95% 97.95% StutBot<br />
27630 root 64 0 1396K 972K RUN 1 76.8H 86.47% 86.47% ee</pre><br />
<br />
Linux:<br />
run [[VPS_Management#vwe|vwe]] to see which VPS’s have high loads. From there run <tt>[[VPS_Management#vp|vp]] <veid></tt> and/or <tt>[[VPS_Management#vt|vt]] <veid></tt> to see what's going on in that system.<br />
[[VPS_Management#vzstat|vzstat]] will also give you a nice picture of whats going on, systems with high numbers in the mlat column are likely culprits.<br />
<br />
examples of out of control procs:<br />
<br />
<pre>12183 nobody 16 0 4916 1348 1340 R 45.5 0.0 4249m httpd<br />
29266 #502 16 0 1852 796 792 R 22.5 0.0 1104m vim<br />
23860 #41 16 0 5472 5472 2076 R 98.9 0.2 31:41 python<br />
19227 bin 19 0 1688 716 652 R 99.9 0.0 321:08 wtrs_ui<br />
7762 apache 16 0 268 236 224 R 85.7 0.0 1010m ptrace<br />
4624 #501 20 0 4304 2400 2044 R 53.6 0.1 284:32 YoSucker<br />
20451 #506 20 0 1876 820 816 R 17.2 0.0 169:35 vim<br />
8834 #514 20 0 900 724 672 R 77.6 0.0 382:30 neostats<br />
31815 apache 14 0 3176 3176 1696 R 74.4 0.1 6:15 counter</pre><br />
<br />
Just kill -9 them and be done with it.<br />
<br />
Also, anytime you see `kmod` or `ptrace` - kill those immediaely no matter how much they are using - they are attempts to exploit the linux ptrace bug. They won't work, but they suck a lot of CPU...<br />
<br />
Also, any other processes that are at 90-100% cpu usage and have been running for any long period of time should be killed except for mysqld processes on FreeBSD. See above.<br />
<br />
However, there is an exception:<br />
<br />
if it is a mysqld, we don't want to kill their database. What you want to do is <tt>[[VPS_Management#jpid|jpid]] <pid></tt> to see who owns it, and then email them the paste containing the instructions for the nanny. Or you can simply do a <tt>kill -1 PID</tt> on the process to restart it.<br />
<br />
=== Load averages jump at night ===<br />
<br />
The load averages on the FreeBSD systems may jump up at night between 1 and 4 am - this is because the backups are running - if this is what is causing a jump in load, you will see processes like `rsync` in top eating a lot of CPU time.<br />
<br />
== check backups ==<br />
<br />
mgmt -> Motnroing -> Backups and make sure every machine was backed up the previous nite. <br />
Also look at df on backup1 and backup2 to make sure no disk is approaching full, though bb should warn us in advance. Please note - errors encountered when a backup script on any of the particular systems run will gnerate an email to support@johncompanies.com so you can know immediately the day after if the directory to be backed up has been moved or no longer exists. A paste exists for this to notify the customer of a non-existant file/dir.<br />
<br />
== check bb for warnings ==<br />
mgmt -> BigBrother<br />
<br />
Some events don't generate pages (on purpose). You will only see them by going to the bb main page.<br />
<br />
== check jail5 for crashed VPSs ==<br />
<br />
On jail5<br />
<br />
notrunning<br />
<br />
To restart a VPS <br />
<br />
vm restart col0XXXX<br />
<br />
== check NetHere ==<br />
<br />
Check the NetHere servers<br />
<br />
=== Mail systems ===<br />
Check for possible SPAMMERs.<br />
<br />
==== Incoming ====<br />
Check mta-1 and mta-2 count of customer logins for possible<br />
customer SPAM compromises.<br />
<br />
login_count /logs/maillog | tail -30<br />
<br />
==== Outgoing ====<br />
<br />
Check outgoing queues on relay-1 and relay-2<br />
<br />
mail_count | tail -30<br />
<br />
To clean up outgoing queues of unwanted SPAM on relay-1 and relay-2.<br />
<br />
mail_cleanup [<sender's domain>]<br />
<br />
=== Nagios ===<br />
<br />
Check for other problems on NetHere servers<br />
<br />
https://nagios.nethere.net<br />
<br />
=== Cacti ===<br />
<br />
Check bandwidth usage on servers<br />
<br />
https://cacti.nethere.net<br />
<br />
= Monthly Tasks =<br />
== rotate pine sent mail (1st of month) ==<br />
On the 1st of the month, before any emails are sent out, quit out of pine, then log back in. Send mail from last month will be archived.<br />
If you mess up and do it on the 3rd (for example), you can go into the previous month's saved email and save emails from the current month into the <tt>sent-mail</tt> (current month) mailbox.<br />
<br />
== b/w caps ==<br />
On the 1st: remove any bwcaps put into the firewall (only really applies if a bwcap was added cause someone went over on b/w):<br />
<br />
ipfw list|grep pipe<br />
ipfw del [each rule listed]<br />
<br />
NOTE: this cronjob on newgateway will do some of that for you, provided you used one of the following pipe #s:<br />
0 0 1 * * /sbin/ipfw del 3 4 5 17331<br />
<br />
-------------------<br />
<br />
We really don’t do this anymore since we have centralized traffic accounting with netflow, but for posterity:<br />
<br />
Make sure all machines reset counters to 0 after midnight on the 1st<br />
Make sure they dumped a counter<br />
<br />
On each jail run:<br />
trafficgather.pl<br />
<br />
And on each virt:<br />
linuxtrafficgather.pl<br />
<br />
== Monthly RAID checks ==<br />
<br />
Every month we check the health of and verfy the parity on all our RAID-based systems.<br />
To facilitate this, we've created a simple script to start the process:<br />
<br />
sh /root/verify.sh<br />
<br />
=== Adaptec controllers ===<br />
Here's some sample output:<br />
<pre><br />
mail /usr/local/www/scripts# sh /root/verify.sh<br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> container list /f<br />
Executing: container list /full=TRUE<br />
Num Total Oth Chunk Scsi Partition<br />
Creation System<br />
Label Type Size Ctr Size Usage B:ID:L Offset:Size State RO Lk Task Done% Ent<br />
Date Time Files<br />
----- ------ ------ --- ------ ------- ------ ------------- ------- -- -- ------- ------ ---<br />
------ -------- ------<br />
0 Mirror 33.9GB Open 0:01:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:32<br />
/dev/aacd0 mirror0 0:00:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:32<br />
<br />
1 Mirror 33.9GB Open 0:02:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:50<br />
/dev/aacd1 mirror1 0:03:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:50<br />
<br />
<br />
AAC0> disk list /f<br />
Executing: disk list /full=TRUE<br />
<br />
B:ID:L Device Type Removable media Vendor-ID Product-ID Rev Blocks Bytes/Bl<br />
ock Usage Shared Rate<br />
------ -------------- --------------- --------- ---------------- ----- --------- --------<br />
--- ---------------- ------ ----<br />
0:00:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:01:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:02:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:03:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
<br />
AAC0> disk show smart<br />
Executing: disk show smart<br />
<br />
Smart Method of Enable<br />
Capable Informational Exception Performance Error<br />
B:ID:L Device Exceptions(MRIE) Control Enabled Count<br />
------ ------- ---------------- --------- ----------- ------<br />
0:00:0 Y 6 Y N 0<br />
0:01:0 Y 6 Y N 0<br />
0:02:0 Y 6 Y N 0<br />
0:03:0 Y 6 Y N 0<br />
0:06:0 N<br />
<br />
AAC0> task list<br />
Executing: task list<br />
<br />
Controller Tasks<br />
<br />
TaskId Function Done% Container State Specific1 Specific2<br />
------ -------- ------- --------- ----- --------- ---------<br />
<br />
No tasks currently running on controller<br />
<br />
AAC0> dia sh hi<br />
Executing: diagnostic show history<br />
No switches specified, defaulting to "/current".<br />
<br />
<br />
<br />
*** HISTORY BUFFER FROM CURRENT CONTROLLER RUN ***<br />
<br />
[00]: GetDiskLogEntry: container - 1, entry return 0<br />
[01]: Container 1 started SCRUB task<br />
[02]: Starting Mirror:1 scrub<br />
[03]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[04]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[05]: UpdateDiskLogIndex - Set - container 0, index 1<br />
[06]: GetDiskLogEntry: container - 0, entry return 1<br />
[07]: Container 0 started SCRUB task<br />
[08]: Starting Mirror:0 scrub<br />
[09]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[10]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[11]: Mirror Scrub Container:1 ErrorsFound:0<br />
[12]: Clear disk log: sector - 80, driveno 2<br />
[13]: Clear disk log: sector - 80, driveno 3<br />
[14]: Container 1 completed SCRUB task:<br />
[15]: Mirror Scrub Container:0 ErrorsFound:0<br />
[16]: Clear disk log: sector - 81, driveno 1<br />
[17]: Clear disk log: sector - 81, driveno 0<br />
[18]: Container 0 completed SCRUB task:<br />
[19]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[20]: GetDiskLogEntry: container - 0, entry return 0<br />
[21]: Container 0 started SCRUB task<br />
[22]: Starting Mirror:0 scrub<br />
[23]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[24]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[25]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[26]: GetDiskLogEntry: container - 1, entry return 1<br />
[27]: Container 1 started SCRUB task<br />
[28]: Starting Mirror:1 scrub<br />
[29]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[30]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[31]: Mirror Scrub Container:1 ErrorsFound:0<br />
[32]: Clear disk log: sector - 81, driveno 2<br />
[33]: Clear disk log: sector - 81, driveno 3<br />
[34]: Container 1 completed SCRUB task:<br />
[35]: Mirror Scrub Container:0 ErrorsFound:0<br />
[36]: Clear disk log: sector - 80, driveno 1<br />
[37]: Clear disk log: sector - 80, driveno 0<br />
[38]: Container 0 completed SCRUB task:<br />
[39]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[40]: GetDiskLogEntry: container - 0, entry return 0<br />
[41]: Container 0 started SCRUB task<br />
[42]: Starting Mirror:0 scrub<br />
[43]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[44]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[45]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[46]: GetDiskLogEntry: container - 1, entry return 1<br />
[47]: Container 1 started SCRUB task<br />
[48]: Starting Mirror:1 scrub<br />
[49]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[50]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[51]: Mirror Scrub Container:1 ErrorsFound:0<br />
[52]: Clear disk log: sector - 81, driveno 2<br />
[53]: Clear disk log: sector - 81, driveno 3<br />
[54]: Container 1 completed SCRUB task:<br />
[55]: Mirror Scrub Container:0 ErrorsFound:0<br />
[56]: Clear disk log: sector - 80, driveno 1<br />
[57]: Clear disk log: sector - 80, driveno 0<br />
[58]: Container 0 completed SCRUB task:<br />
[59]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[60]: GetDiskLogEntry: container - 0, entry return 0<br />
[61]: Container 0 started SCRUB task<br />
[62]: Starting Mirror:0 scrub<br />
[63]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[64]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[65]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[66]: GetDiskLogEntry: container - 1, entry return 1<br />
[67]: Container 1 started SCRUB task<br />
[68]: Starting Mirror:1 scrub<br />
[69]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[70]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[71]: Mirror Scrub Container:1 ErrorsFound:0<br />
[72]: Clear disk log: sector - 81, driveno 2<br />
[73]: Clear disk log: sector - 81, driveno 3<br />
[74]: Container 1 completed SCRUB task:<br />
[75]: Mirror Scrub Container:0 ErrorsFound:0<br />
[76]: Clear disk log: sector - 80, driveno 1<br />
[77]: Clear disk log: sector - 80, driveno 0<br />
[78]: Container 0 completed SCRUB task:<br />
[79]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[80]: GetDiskLogEntry: container - 0, entry return 0<br />
[81]: Container 0 started SCRUB task<br />
[82]: Starting Mirror:0 scrub<br />
[83]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[84]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[85]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[86]: GetDiskLogEntry: container - 1, entry return 1<br />
[87]: Container 1 started SCRUB task<br />
[88]: Starting Mirror:1 scrub<br />
[89]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[90]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[91]: Mirror Scrub Container:1 ErrorsFound:0<br />
[92]: Clear disk log: sector - 81, driveno 2<br />
[93]: Clear disk log: sector - 81, driveno 3<br />
[94]: Container 1 completed SCRUB task:<br />
[95]: Mirror Scrub Container:0 ErrorsFound:0<br />
[96]: Clear disk log: sector - 80, driveno 1<br />
[97]: Clear disk log: sector - 80, driveno 0<br />
[98]: Container 0 completed SCRUB task:<br />
[99]:<br />
<br />
========================<br />
History Output Complete.<br />
<br />
AAC0><br />
AAC0> exit<br />
Executing: exit<br />
<br />
press enter when ready to run verify <INS><br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> contai scr 0<br />
Executing: container scrub 0<br />
<br />
AAC0> contai scr 1<br />
Executing: container scrub 1<br />
<br />
AAC0> exit<br />
Executing: exit<br />
<br />
when done run: <br />
<br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
<br />
<br />
Nov 1 10:32:46 mail /kernel: aac0: **Monitor** Container 0 started SCRUB task<br />
Nov 1 10:32:47 mail /kernel: aac0: **Monitor** Container 1 started SCRUB task<br />
<br />
</pre><br />
<br />
Here's an analysis of what we're seeing and what we're looking for:<br />
<br />
<pre><br />
AAC0> container list /f<br />
Executing: container list /full=TRUE<br />
Num Total Oth Chunk Scsi Partition<br />
Creation System<br />
Label Type Size Ctr Size Usage B:ID:L Offset:Size State RO Lk Task Done% Ent<br />
Date Time Files<br />
----- ------ ------ --- ------ ------- ------ ------------- ------- -- -- ------- ------ ---<br />
------ -------- ------<br />
0 Mirror 33.9GB Open 0:01:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:32<br />
/dev/aacd0 mirror0 0:00:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:32<br />
<br />
1 Mirror 33.9GB Open 0:02:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:50<br />
/dev/aacd1 mirror1 0:03:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:50<br />
</pre><br />
<br />
This is showing you the health of the arrays. You're looking for ''Normal'' under the State column, and the absence of a ! in the sector size - sometimes, you'll see this:<br />
64.0KB!33.9GB <br />
<br />
That indicates a problem.<br />
<br />
<pre><br />
AAC0> disk show smart<br />
Executing: disk show smart<br />
<br />
Smart Method of Enable<br />
Capable Informational Exception Performance Error<br />
B:ID:L Device Exceptions(MRIE) Control Enabled Count<br />
------ ------- ---------------- --------- ----------- ------<br />
0:00:0 Y 6 Y N 0<br />
0:01:0 Y 6 Y N 0<br />
0:02:0 Y 6 Y N 0<br />
0:03:0 Y 6 Y N 0<br />
0:06:0 N<br />
</pre><br />
<br />
This shows you a SMART report output. Looking for values in the Error Count column.<br />
<pre><br />
AAC0> task list<br />
Executing: task list<br />
<br />
Controller Tasks<br />
<br />
TaskId Function Done% Container State Specific1 Specific2<br />
------ -------- ------- --------- ----- --------- ---------<br />
<br />
No tasks currently running on controller<br />
</pre><br />
<br />
Look for absence of tasks running- a bad thing would be to see a rebuild or verify running when you didn't initiate it.<br />
<br />
With the history output, you're looking for any anomalies or events since the last time a verify was run. If you see a drive with lots of problems, you may want to take backups before allowing the verify to run since it could replicate errors onto the good drive.<br />
<br />
After you see the history output, it will prompt you to press enter to run the verify. If you're happy with all the output you're seeing- mirror is healthy, history looks good, it's safe to proceed. Otherwise ^C to exit. After hitting enter it will start the verify and start to tail the messages log file (so you can easily see when the verify is complete). Here's what that'll look like:<br />
<br />
<pre><br />
Nov 1 14:38:08 mail /kernel: aac0: **Monitor** Container 1 completed SCRUB task:<br />
Nov 1 14:46:45 mail /kernel: aac0: **Monitor** Container 0 completed SCRUB task:<br />
</pre><br />
<br />
So, putting it all together, after hitting enter to start the verify, you'll see:<br />
<br />
<pre><br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> contai scr 0<br />
Executing: container scrub 0<br />
<br />
AAC0> contai scr 1<br />
Executing: container scrub 1<br />
<br />
AAC0> exit<br />
Executing: exit<br />
<br />
when done run: <br />
<br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
<br />
<br />
Nov 1 10:32:46 mail /kernel: aac0: **Monitor** Container 0 started SCRUB task<br />
Nov 1 10:32:47 mail /kernel: aac0: **Monitor** Container 1 started SCRUB task<br />
<br />
</pre><br />
<br />
When the scrub(s) (verify) are complete - if the server has multiple logical drives, it will run both in parallel - you should exit the tail of the log file (^C) and run:<br />
<br />
<pre><br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
</pre><br />
<br />
Which will show you the diagnostic history, you're looking for the results of the most recent scrub:<br />
<br />
<pre><br />
[100]: Mirror Scrub Container:1 ErrorsFound:0<br />
[101]: Clear disk log: sector - 81, driveno 2<br />
[102]: Clear disk log: sector - 81, driveno 3<br />
[103]: Container 1 completed SCRUB task:<br />
[104]: Mirror Scrub Container:0 ErrorsFound:0<br />
[105]: Clear disk log: sector - 80, driveno 1<br />
[106]: Clear disk log: sector - 80, driveno 0<br />
[107]: Container 0 completed SCRUB task:<br />
</pre><br />
<br />
^C to exit the RAID CLI.<br />
<br />
If you see:<br />
<br />
[104]: Mirror Scrub Container:0 ErrorsFound:5<br />
<br />
You'll want to rerun the verify on that drive till it shows 0, or perhaps replace the drive- you should be able to see from the output which drive had the problem.<br />
<br />
Depending on the size and how busy the drive is, the verify can take anywhere from an hour to the better part of a day.<br />
<br />
You will notice that the diagnostic history is not shown on our modern adaptec cards (i.e. any adaptec card not in a Dell 2450). The reason for this is the history is never cleared, so there's simply too much data to show and it just crashes the CLI. So, don't bother trying to see it...which does make it hard to see if there are problems going on, so you just need to watch the scrub to see it goes to 100%. You will also notice that on some servers there's no tail of messages. Again, this is cause no data is shown there about the completion of the scrub. The thing to do here is to go into the CLI and continue to <tt>show tasks</tt> to monitor scrub progress.<br />
<br />
See [[RAIC_CLI#Adaptec|Adaptec RAID CLI Reference]] for more details on how to use the CLI<br />
<br />
=== DELL (LSI-based) SAS controllers ===<br />
<br />
Here's what the output looks like when running verify.sh on a LSI-based card:<br />
<br />
<pre><br />
jail2 /mnt/data2# sh /root/verify.sh<br />
<br />
Adapter #0<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 0<br />
Device Id: 0<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018396142<br />
SAS Address(1): 0x0<br />
Connected Port Number: 0(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DESN<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 1<br />
Device Id: 1<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018395db2<br />
SAS Address(1): 0x0<br />
Connected Port Number: 1(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DERV<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 2<br />
Device Id: 2<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50006eece89<br />
SAS Address(1): 0x0<br />
Connected Port Number: 2(path0)<br />
Inquiry Data: SEAGATE ST3300555SS T2113LM4BFBZ<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 3<br />
Device Id: 3<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50006eee035<br />
SAS Address(1): 0x0<br />
Connected Port Number: 3(path0)<br />
Inquiry Data: SEAGATE ST3300555SS T2113LM4BGF7<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 4<br />
Device Id: 4<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50004bd7ea5<br />
SAS Address(1): 0x0<br />
Connected Port Number: 4(path0)<br />
Inquiry Data: SEAGATE ST3300656SS HS093QP0G8SW<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 5<br />
Device Id: 5<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e01f1c4112<br />
SAS Address(1): 0x0<br />
Connected Port Number: 5(path0)<br />
Inquiry Data: FUJITSU MBA3300RC D306BJ15P9201W06<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 0 (Target Id: 0)<br />
Name:<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:139392MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 1 (Target Id: 1)<br />
Name:MIRROR1<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 2 (Target Id: 2)<br />
Name:MIRROR2<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
Battery FRU : N/A<br />
Battery Warning : Enabled<br />
Memory Correctable Errors : 0<br />
Memory Uncorrectable Errors : 0<br />
BBU : Present<br />
BBU : Yes<br />
Cache When BBU Bad : Disabled<br />
press enter when ready to run verify<br />
<br />
</pre><br />
<br />
Before pressing enter, here's what we're looking for:<br />
<br />
<pre><br />
Enclosure Device ID: 32<br />
Slot Number: 0<br />
Device Id: 0<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018396142<br />
SAS Address(1): 0x0<br />
Connected Port Number: 0(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DESN<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
</pre><br />
<br />
This is the output shown for each physical drive in the system. We're looking to confirm it's Firmware state is ''Online'', and Media Error Count, Other Error Count, and Predictive Failure Count are all zero (or near zero).<br />
<br />
<pre><br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 1 (Target Id: 1)<br />
Name:MIRROR1<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
</pre><br />
<br />
This is the output for each logical drive. We're looking for State ''Optimal''. Also confirm <tt>Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU</tt><br />
<br />
<pre><br />
Exit Code: 0x00<br />
Battery FRU : N/A<br />
Battery Warning : Enabled<br />
Memory Correctable Errors : 0<br />
Memory Uncorrectable Errors : 0<br />
BBU : Present<br />
BBU : Yes<br />
Cache When BBU Bad : Disabled<br />
</pre><br />
<br />
Confirm that the battery is present and error-free.<br />
<br />
If all that checks out, you're ready to proceed with the verify. After pressing enter, the verify is started and here's what you see:<br />
<pre><br />
Start Check Consistency on Virtual Drive 0 (target id: 0) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Start Check Consistency on Virtual Drive 1 (target id: 1) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Start Check Consistency on Virtual Drive 2 (target id: 2) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Check Consistency<br />
<br />
Progress of Virtual Drives...<br />
<br />
Virtual Drive # Percent Complete Time Elps<br />
0 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:03<br />
1 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:02<br />
2 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:01<br />
<br />
Press <ESC> key to quit...<br />
</pre><br />
<br />
The progress for each drive is displayed until all drives have completed the verify. We just want to make sure that each drive goes to completion. No followup is needed...though there probably is a log or history where we can get more info.<br />
<br />
You will notice that jail7 does not run a verify- that's on purpose. The last time we tried this it crashed the system. So, this must be run from the BIOS (take the system offline for a couple hours).<br />
<br />
See [[RAIC_CLI#LSI|LSI RAID CLI Reference]] for more details on how to use the CLI<br />
<br />
=== LSI-based controllers (megaraid) ===<br />
<br />
There is a CLI for this however it's easier to do this with a curses GUI app: megaraid<br />
<br />
Currently only on these servers: virt15, virt16, and firewall2<br />
<br />
To run:<br />
<br />
# cd /usr/local/sbin/; megamgr<br />
<br />
Main menu:<br />
<pre>²ÚÄÄManagement MenuÄÄ¿²<br />
²³ Configure ³²<br />
²³ Initialize ³²<br />
²³ Objects ³²<br />
²³ Rebuild ³²<br />
²³ Check Consistency ³²<br />
²³ Advanced Menu ³²<br />
²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²</pre><br />
<br />
Before you check consistency, make sure the arrays are healthy.<br />
<br />
Objects -> Physical Drive<br />
<br />
Then look to make sure they're all <tt>ONLIN</tt><br />
<pre><br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄObjects - PHYSICAL DRIVE SELECTION MENUÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²<br />
²²²²²²²²³ ³²²²²²²<br />
²²²²²²²²³ Channel-1 ³²²²²²²<br />
²ÚÄÄMana³ ID ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»x ³²²²²²²<br />
²³ Confi³ 0º* ONLIN A01-01º ³²²²²²²<br />
²³ Initi³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²³ Objec³ 1º* ONLIN A01-02º ³²²²²²²<br />
²³ Rebui³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²³ Check³ 2º* ONLIN A02-01º ³²²²²²²<br />
²³ Advan³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²ÀÄÄÄÄÄij 3º* ONLIN A02-02º ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 4º* ONLIN A03-01º ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 5º* ONLIN A03-02ºþ ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 6º* º ³²²²²²²<br />
²²²²²²²²³ ÈÍÍÍÍÍÍÍÍÍÍÍÍÍͼx ³²²²²²²<br />
²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²<br />
²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²²²<br />
²²²²²²²²³Ch-1 ID-5 DISK 140013MB SEAGATE ST3146707LC 0003 ³²²²²²²²²<br />
²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²<br />
</pre><br />
<br />
<br />
Once that's done, hit escape once then the back arror to move back to the Objects menu.<br />
So you select Objects -> Logical Drive -> Logical Drive 1 -> Check Consistency -> YES<br />
<br />
<pre><br />
<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÚÄLogical Drives(02)Ä¿²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ Logical Drive 1 ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²ÚÄÄÄÄObjectsÄÄij Logical Drive 2 ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²ÚÄÄManagemen³ Adapter ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Configure ³ Logical Drive ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Initialize³ Physical Drive ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Objects ³ Channel ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Rebuild ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Check Consistency ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Advanced Menu ³²²²²²²²ÚÄÄÄÄLogical Drive 1ÄÄÄÄÄ¿²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²³ Initialize ÚÄCheck Consistency-1 ?Ä¿²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ Check Consiste³ YES ³²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ View/Update Pa³ NO ³²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³Select YES Or NO³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
</pre><br />
<br />
Then watch the progress. When done, escape back to Logical Drive then repeat for Logical Drive 2. If you ^C or accidentally escape out, you can come back in running the same commands and watch the progress again (it won't restart).<br />
<br />
You can exit megamgr by escaping out or ^C<br />
<br />
=== 3ware ===<br />
<br />
We are using 3ware controllers on backup1 & backup2. Running the verify script will give you different output based on the type of controller:<br />
<br />
<pre><br />
backup2 /d2# sh /root/verify.sh<br />
Controller: c0<br />
-------------<br />
Driver: 1.50.01.002<br />
Model: 7500-8<br />
FW: FE7X 1.05.00.068<br />
BIOS: BE7X 1.08.00.048<br />
Monitor: ME7X 1.01.00.040<br />
Serial #: F11605A3180172<br />
PCB: Rev3<br />
PCHIP: 1.30-33<br />
ACHIP: 3.20<br />
<br />
<br />
# of units: 3<br />
Unit 0: JBOD 186.31 GB ( 390721968 blocks): OK<br />
Unit 1: RAID 5 465.77 GB ( 976790016 blocks): DEGRADED<br />
Unit 5: RAID 5 698.65 GB ( 1465185024 blocks): DEGRADED<br />
<br />
# of ports: 8<br />
Port 0: WDC WD2000JB-00KFA0 WD-WCAMT1451690 186.31 GB (390721968 blocks): OK(unit 0)<br />
Port 1: WDC WD2500JB-00GVC0 WD-WCAL78219488 232.88 GB (488397168 blocks): OK(unit 1)<br />
Port 2: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT)<br />
Port 3: WDC WD2500JB-00GVC0 WD-WMAL73882417 232.88 GB (488397168 blocks): OK(unit 1)<br />
Port 4: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT)<br />
Port 5: WDC WD2500JB-00GVA0 WD-WMAL71338097 232.88 GB (488397168 blocks): OK(unit 5)<br />
Port 6: WDC WD2500JB-32EVA0 WD-WMAEH1301595 232.88 GB (488397168 blocks): OK(unit 5)<br />
Port 7: WDC WD2500JB-00GVC0 WD-WCAL78165566 232.88 GB (488397168 blocks): OK(unit 5)<br />
Controller: c1<br />
-------------<br />
Driver: 1.50.01.002<br />
Model: 7500-8<br />
FW: FE7X 1.05.00.068<br />
BIOS: BE7X 1.08.00.048<br />
Monitor: ME7X 1.01.00.040<br />
Serial #: F11605A3180167<br />
PCB: Rev3<br />
PCHIP: 1.30-33<br />
ACHIP: 3.20<br />
<br />
<br />
# of units: 2<br />
Unit 0: RAID 5 698.65 GB ( 1465185024 blocks): OK<br />
Unit 4: RAID 5 698.65 GB ( 1465185024 blocks): OK<br />
<br />
# of ports: 8<br />
Port 0: WDC WD2500JB-00GVA0 WD-WMAL71301258 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 1: WDC WD2500JB-00GVA0 WD-WMAL71322705 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 2: WDC WD2500JB-00GVA0 WD-WMAL71945050 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 3: WDC WD2500JB-00GVA0 WD-WMAL71316201 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 4: WDC WD2500JB-00GVC0 WD-WCAL78323749 232.88 GB (488397168 blocks): OK(unit 4)<br />
Port 5: WDC WD3200AAJB-00J3A0 WD-WCAV2V689068 298.09 GB (625142448 blocks): OK(unit 4)<br />
Port 6: WDC WD2500JB-00GVC0 WD-WCAL78234420 232.88 GB (488397168 blocks): OK(unit 4)<br />
Port 7: WDC WD2500JB-00GVC0 WD-WCAL78592213 232.88 GB (488397168 blocks): OK(unit 4)<br />
backup2 /d2#<br />
</pre><br />
<br />
On backup2 look for all ok, no verify.<br />
<br />
<pre><br />
[root@backup3 ~]# sh /root/verify.sh<br />
/c2 Driver Version = 1.26.02.002<br />
/c2 Model = 8006-2LP<br />
/c2 Available Memory = 512KB<br />
/c2 Firmware Version = FE8S 1.05.00.068<br />
/c2 Bios Version = BE7X 1.08.00.048<br />
/c2 Boot Loader Version = ME7X 1.01.00.040<br />
/c2 Serial Number = L018501C6481395<br />
/c2 PCB Version = Rev5<br />
/c2 PCHIP Version = 1.30-66<br />
/c2 ACHIP Version = 3.20<br />
/c2 Total Optimal Units = 1<br />
/c2 Not Optimal Units = 0<br />
<br />
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u1 RAID-1 OK - - - 931.512 ON -<br />
<br />
Port Status Unit Size Blocks Serial<br />
---------------------------------------------------------------<br />
p0 OK u1 931.51 GB 1953525168 WD-WMAW31148820<br />
p1 OK u1 931.51 GB 1953525168 WD-WCATR0277515<br />
<br />
<br />
Ctl Date Severity Alarm Message<br />
------------------------------------------------------------------------------<br />
<br />
Sending start verify message to /c2/u1 ... Done.<br />
<br />
<br />
when done run:<br />
tw_cli /c2 show alarms<br />
<br />
[root@backup3 ~]#<br />
</pre><br />
<br />
Automatically starts the verify, just run <tt>tw_cli /c2 show alarms</tt> as instructed to see the results of the verify.<br />
<br />
<pre><br />
[root@backup1 /data/deprecated]# sh /root/verify.sh<br />
/c0 Driver Version = 2.26.02.010<br />
/c0 Model = 9650SE-8LPML<br />
/c0 Available Memory = 224MB<br />
/c0 Firmware Version = FE9X 4.06.00.004<br />
/c0 Bios Version = BE9X 4.05.00.015<br />
/c0 Boot Loader Version = BL9X 3.08.00.001<br />
/c0 Serial Number = L326025A8270177<br />
/c0 PCB Version = Rev 032<br />
/c0 PCHIP Version = 2.00<br />
/c0 ACHIP Version = 1.90<br />
/c0 Number of Ports = 8<br />
/c0 Number of Drives = 6<br />
/c0 Number of Units = 1<br />
/c0 Total Optimal Units = 1<br />
/c0 Not Optimal Units = 0<br />
/c0 JBOD Export Policy = off<br />
/c0 Disk Spinup Policy = 1<br />
/c0 Spinup Stagger Time Policy (sec) = 1<br />
/c0 Auto-Carving Policy = off<br />
/c0 Auto-Carving Size = 2048 GB<br />
/c0 Auto-Rebuild Policy = on<br />
/c0 Controller Bus Type = PCIe<br />
/c0 Controller Bus Width = 1 lane<br />
/c0 Controller Bus Speed = 2.5 Gbps/lane<br />
<br />
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u0 RAID-5 OK - - 64K 4656.56 ON ON<br />
<br />
Port Status Unit Size Blocks Serial<br />
---------------------------------------------------------------<br />
p0 OK u0 931.51 GB 1953525168 9QJ1Y017<br />
p1 DEVICE-ERROR u0 931.51 GB 1953525168 9QJ1ZN07<br />
p2 OK u0 931.51 GB 1953525168 9QJ2XK1R<br />
p3 OK u0 931.51 GB 1953525168 9QJ2010B<br />
p4 OK u0 1.36 TB 2930277168 6XW0L36T<br />
p5 OK u0 931.51 GB 1953525168 WD-WMATV2444836<br />
p6 NOT-PRESENT - - - -<br />
p7 NOT-PRESENT - - - -<br />
<br />
<br />
Ctl Date Severity Alarm Message<br />
------------------------------------------------------------------------------<br />
c0 [Sat May 12 11:27:15 2012] WARNING Sector repair completed: port=0, LBA=0x6AE571C<br />
c0 [Sat May 12 19:16:21 2012] WARNING Sector repair completed: port=1, LBA=0x40E62A23<br />
c0 [Sat May 12 21:40:56 2012] INFO Verify completed: unit=0<br />
c0 [Mon May 14 00:53:53 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFA7<br />
c0 [Mon May 14 00:58:21 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAA<br />
c0 [Mon May 14 04:35:13 2012] WARNING Sector repair completed: port=0, LBA=0x8FEF2CF<br />
c0 [Mon May 14 04:38:22 2012] WARNING Sector repair completed: port=0, LBA=0x8FEF2D1<br />
c0 [Tue May 15 22:53:46 2012] WARNING Sector repair completed: port=0, LBA=0x13C2622<br />
c0 [Wed May 16 00:39:31 2012] WARNING Sector repair completed: port=0, LBA=0x365A67F<br />
c0 [Wed May 16 00:39:37 2012] WARNING Sector repair completed: port=0, LBA=0x365A685<br />
c0 [Wed May 16 00:47:18 2012] WARNING Sector repair completed: port=0, LBA=0x365A687<br />
c0 [Sat May 19 00:01:44 2012] INFO Verify started: unit=0<br />
c0 [Sat May 19 04:46:20 2012] WARNING Sector repair completed: port=0, LBA=0x365A68E<br />
c0 [Sat May 19 13:37:06 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAC<br />
c0 [Sat May 19 13:37:28 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAE<br />
c0 [Sat May 19 13:37:47 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFB1<br />
c0 [Sat May 19 13:38:00 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFB3<br />
c0 [Sat May 19 21:47:45 2012] INFO Verify completed: unit=0<br />
c0 [Wed May 23 12:21:41 2012] INFO Cache synchronization completed: unit=0<br />
c0 [Fri May 25 00:08:19 2012] WARNING Sector repair completed: port=0, LBA=0x12DA76C<br />
c0 [Fri May 25 00:08:34 2012] WARNING Sector repair completed: port=0, LBA=0x12E4901<br />
c0 [Fri May 25 00:09:33 2012] WARNING Sector repair completed: port=0, LBA=0x12DA773<br />
c0 [Fri May 25 00:39:12 2012] WARNING Sector repair completed: port=0, LBA=0x42C597B<br />
c0 [Sat May 26 00:01:45 2012] INFO Verify started: unit=0<br />
c0 [Sat May 26 00:42:05 2012] WARNING Sector repair completed: port=1, LBA=0x323C1AC<br />
c0 [Sat May 26 00:51:43 2012] WARNING Sector repair completed: port=1, LBA=0x323C1AE<br />
c0 [Sat May 26 01:54:21 2012] WARNING Sector repair completed: port=1, LBA=0x2F0D302<br />
c0 [Sat May 26 02:06:38 2012] WARNING Sector repair completed: port=0, LBA=0x12DA777<br />
c0 [Sat May 26 02:07:21 2012] WARNING Sector repair completed: port=0, LBA=0x12E48FE<br />
c0 [Sat May 26 04:20:00 2012] WARNING Sector repair completed: port=1, LBA=0x2F0D306<br />
c0 [Sat May 26 04:32:58 2012] WARNING Sector repair completed: port=1, LBA=0x323C1B1<br />
c0 [Sat May 26 04:33:21 2012] WARNING Sector repair completed: port=1, LBA=0x323C1B3<br />
c0 [Sat May 26 04:33:44 2012] WARNING Sector repair completed: port=1, LBA=0x323C1BA<br />
c0 [Sat May 26 05:24:07 2012] WARNING Sector repair completed: port=1, LBA=0x3F83862<br />
c0 [Sat May 26 05:25:09 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 06:08:13 2012] WARNING Sector repair completed: port=0, LBA=0x4CDC6A2<br />
c0 [Sat May 26 09:49:35 2012] WARNING Sector repair completed: port=1, LBA=0x6CACD4A<br />
c0 [Sat May 26 18:10:44 2012] WARNING Sector repair completed: port=1, LBA=0x18F425EA<br />
c0 [Sat May 26 19:45:40 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:22:52 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:15 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:22 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:35 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:41 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:49 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:57 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:24:02 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:54:41 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 22:00:30 2012] INFO Verify completed: unit=0<br />
c0 [Sat Jun 2 00:01:43 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 2 00:30:17 2012] WARNING Sector repair completed: port=0, LBA=0x2B911E4<br />
c0 [Sat Jun 2 00:50:57 2012] WARNING Sector repair completed: port=0, LBA=0x5A807CA6<br />
c0 [Sat Jun 2 04:13:13 2012] WARNING Sector repair completed: port=0, LBA=0x2D18291<br />
c0 [Sat Jun 2 04:13:35 2012] WARNING Sector repair completed: port=0, LBA=0x2D1829F<br />
c0 [Sat Jun 2 21:48:02 2012] INFO Verify completed: unit=0<br />
c0 [Mon Jun 4 04:40:34 2012] WARNING Sector repair completed: port=1, LBA=0x4AF8098F<br />
c0 [Tue Jun 5 00:28:19 2012] WARNING Sector repair completed: port=1, LBA=0x263C5CD<br />
c0 [Tue Jun 5 00:33:06 2012] WARNING Sector repair completed: port=1, LBA=0x263C5CF<br />
c0 [Thu Jun 7 00:34:27 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B5F<br />
c0 [Thu Jun 7 00:38:50 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B61<br />
c0 [Fri Jun 8 00:07:13 2012] WARNING Sector repair completed: port=0, LBA=0xC131F6B<br />
c0 [Sat Jun 9 00:01:41 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 9 00:29:11 2012] WARNING Sector repair completed: port=0, LBA=0x6C7614D<br />
c0 [Sat Jun 9 00:38:25 2012] WARNING Sector repair completed: port=0, LBA=0x6C76152<br />
c0 [Sat Jun 9 04:02:30 2012] WARNING Sector repair completed: port=1, LBA=0x263C5D1<br />
c0 [Sat Jun 9 04:02:52 2012] WARNING Sector repair completed: port=1, LBA=0x263C5D3<br />
c0 [Sat Jun 9 04:07:32 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E12<br />
c0 [Sat Jun 9 04:07:57 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E15<br />
c0 [Sat Jun 9 04:08:16 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E17<br />
c0 [Sat Jun 9 04:08:45 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E19<br />
c0 [Sat Jun 9 04:15:04 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B64<br />
c0 [Sat Jun 9 04:15:26 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B66<br />
c0 [Sat Jun 9 04:15:45 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B68<br />
c0 [Sat Jun 9 04:15:59 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B6C<br />
c0 [Sat Jun 9 04:16:13 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B6E<br />
c0 [Sat Jun 9 21:48:52 2012] INFO Verify completed: unit=0<br />
c0 [Thu Jun 14 00:40:10 2012] WARNING Sector repair completed: port=0, LBA=0x334F14B<br />
c0 [Sat Jun 16 00:01:38 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 16 21:16:19 2012] INFO Verify completed: unit=0<br />
c0 [Tue Jun 19 02:03:43 2012] WARNING Sector repair completed: port=1, LBA=0xFE41EAD<br />
c0 [Wed Jun 20 02:30:02 2012] WARNING Sector repair completed: port=1, LBA=0xD99145C<br />
c0 [Sat Jun 23 00:01:36 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 23 04:27:04 2012] WARNING Sector repair completed: port=1, LBA=0x2FAD311<br />
c0 [Sat Jun 23 06:52:38 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC8D<br />
c0 [Sat Jun 23 06:53:03 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC91<br />
c0 [Sat Jun 23 06:53:21 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC94<br />
c0 [Sat Jun 23 17:00:22 2012] WARNING Sector repair completed: port=1, LBA=0xF9AC7C9<br />
c0 [Sat Jun 23 21:15:19 2012] INFO Verify completed: unit=0<br />
c0 [Sat Jun 30 00:01:34 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 30 05:24:13 2012] WARNING Sector repair completed: port=0, LBA=0x3FAA9E7<br />
c0 [Sat Jun 30 14:49:39 2012] WARNING Sector repair completed: port=1, LBA=0x869931C<br />
c0 [Sat Jun 30 21:31:05 2012] INFO Verify completed: unit=0<br />
c0 [Tue Jul 3 03:40:25 2012] WARNING Sector repair completed: port=1, LBA=0xD36C7F7<br />
c0 [Fri Jul 6 02:50:18 2012] WARNING Sector repair completed: port=1, LBA=0x3562470<br />
c0 [Fri Jul 6 22:18:26 2012] WARNING Sector repair completed: port=1, LBA=0x3563173<br />
c0 [Sat Jul 7 00:01:31 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 7 00:50:16 2012] WARNING Sector repair completed: port=0, LBA=0x76EE88<br />
c0 [Sat Jul 7 00:50:39 2012] WARNING Sector repair completed: port=0, LBA=0x76EE8F<br />
c0 [Sat Jul 7 21:39:36 2012] INFO Verify completed: unit=0<br />
c0 [Sun Jul 8 02:51:05 2012] WARNING Sector repair completed: port=0, LBA=0x67759D<br />
c0 [Sun Jul 8 02:53:55 2012] WARNING Sector repair completed: port=0, LBA=0x67759B<br />
c0 [Tue Jul 10 16:17:21 2012] WARNING Sector repair completed: port=0, LBA=0x15C8C695<br />
c0 [Wed Jul 11 22:51:22 2012] WARNING Sector repair completed: port=1, LBA=0x355BBD0<br />
c0 [Sat Jul 14 00:01:28 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 14 01:33:40 2012] WARNING Sector repair completed: port=1, LBA=0x1333BCF4<br />
c0 [Sat Jul 14 03:36:23 2012] WARNING Sector repair completed: port=1, LBA=0x2174773<br />
c0 [Sat Jul 14 11:26:44 2012] WARNING Sector repair completed: port=0, LBA=0x7429AB7<br />
c0 [Sat Jul 14 16:53:50 2012] WARNING Sector repair completed: port=1, LBA=0xA17EB3F<br />
c0 [Sat Jul 14 21:19:25 2012] INFO Verify completed: unit=0<br />
c0 [Wed Jul 18 05:08:47 2012] WARNING Sector repair completed: port=1, LBA=0x17D62EDC<br />
c0 [Wed Jul 18 05:14:15 2012] WARNING Sector repair completed: port=1, LBA=0x17D62EE1<br />
c0 [Thu Jul 19 03:24:59 2012] WARNING Sector repair completed: port=0, LBA=0x7733C3D<br />
c0 [Thu Jul 19 03:25:20 2012] WARNING Sector repair completed: port=0, LBA=0x773CEA5<br />
c0 [Thu Jul 19 03:28:16 2012] WARNING Sector repair completed: port=0, LBA=0x7733C42<br />
c0 [Thu Jul 19 03:28:41 2012] WARNING Sector repair completed: port=0, LBA=0x773CEAF<br />
c0 [Sat Jul 21 00:01:26 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 21 03:07:30 2012] WARNING Sector repair completed: port=1, LBA=0x1CC6936<br />
c0 [Sat Jul 21 03:07:52 2012] WARNING Sector repair completed: port=1, LBA=0x1CC6938<br />
c0 [Sat Jul 21 03:08:11 2012] WARNING Sector repair completed: port=1, LBA=0x1CC693A<br />
c0 [Sat Jul 21 16:43:56 2012] WARNING Sector repair completed: port=0, LBA=0xD04C914<br />
c0 [Sat Jul 21 16:45:31 2012] WARNING Sector repair completed: port=1, LBA=0xD456973<br />
c0 [Sat Jul 21 21:14:29 2012] INFO Verify completed: unit=0<br />
c0 [Wed Jul 25 03:37:25 2012] WARNING Sector repair completed: port=0, LBA=0x1F8E6C43<br />
c0 [Sat Jul 28 00:01:24 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 28 01:45:27 2012] WARNING Sector repair completed: port=0, LBA=0x11584AD<br />
c0 [Sat Jul 28 18:54:25 2012] WARNING Sector repair completed: port=1, LBA=0x447C3E6C<br />
c0 [Sat Jul 28 21:13:46 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 1 03:20:11 2012] WARNING Sector repair completed: port=0, LBA=0x805FEF<br />
c0 [Fri Aug 3 00:50:03 2012] WARNING Sector repair completed: port=0, LBA=0xCED0ACA<br />
c0 [Sat Aug 4 00:01:22 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 4 00:52:51 2012] WARNING Sector repair completed: port=0, LBA=0x805FF3<br />
c0 [Sat Aug 4 00:53:14 2012] WARNING Sector repair completed: port=0, LBA=0x805FF5<br />
c0 [Sat Aug 4 00:53:33 2012] WARNING Sector repair completed: port=0, LBA=0x805FF7<br />
c0 [Sat Aug 4 00:53:47 2012] WARNING Sector repair completed: port=0, LBA=0x805FF9<br />
c0 [Sat Aug 4 00:54:00 2012] WARNING Sector repair completed: port=0, LBA=0x805FFB<br />
c0 [Sat Aug 4 00:54:14 2012] WARNING Sector repair completed: port=0, LBA=0x805FFD<br />
c0 [Sat Aug 4 00:54:27 2012] WARNING Sector repair completed: port=0, LBA=0x805FFF<br />
c0 [Sat Aug 4 04:43:12 2012] WARNING Sector repair completed: port=1, LBA=0x16974289<br />
c0 [Sat Aug 4 04:58:17 2012] WARNING Sector repair completed: port=1, LBA=0x1697428E<br />
c0 [Sat Aug 4 20:54:53 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 8 03:21:55 2012] ERROR Drive timeout detected: port=1<br />
c0 [Wed Aug 8 15:31:44 2012] WARNING Sector repair completed: port=0, LBA=0x1A366CD3<br />
c0 [Sat Aug 11 00:01:21 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 11 20:40:51 2012] INFO Verify completed: unit=0<br />
c0 [Thu Aug 16 05:10:55 2012] WARNING Sector repair completed: port=0, LBA=0x1C22593<br />
c0 [Sat Aug 18 00:01:18 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 18 03:00:20 2012] WARNING Sector repair completed: port=0, LBA=0x1C225A5<br />
c0 [Sat Aug 18 03:43:00 2012] WARNING Sector repair completed: port=1, LBA=0x23EE91E<br />
c0 [Sat Aug 18 03:43:23 2012] WARNING Sector repair completed: port=1, LBA=0x23EE920<br />
c0 [Sat Aug 18 17:00:06 2012] WARNING Sector repair completed: port=1, LBA=0x137D066A<br />
c0 [Sat Aug 18 17:00:29 2012] WARNING Sector repair completed: port=1, LBA=0x137D066D<br />
c0 [Sat Aug 18 21:13:01 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 22 01:36:08 2012] WARNING Sector repair completed: port=0, LBA=0x2560A0F<br />
c0 [Wed Aug 22 01:37:42 2012] WARNING Sector repair completed: port=0, LBA=0x2560A13<br />
c0 [Fri Aug 24 04:01:36 2012] WARNING Sector repair completed: port=1, LBA=0x55C1A5DF<br />
c0 [Fri Aug 24 05:02:06 2012] WARNING Sector repair completed: port=1, LBA=0xCE3378A<br />
c0 [Sat Aug 25 00:01:17 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 25 00:31:06 2012] WARNING Sector repair completed: port=1, LBA=0x50F65D<br />
c0 [Sat Aug 25 00:39:52 2012] WARNING Sector repair completed: port=0, LBA=0x678FF4<br />
c0 [Sat Aug 25 03:43:15 2012] WARNING Sector repair completed: port=0, LBA=0x2560A15<br />
c0 [Sat Aug 25 03:43:39 2012] WARNING Sector repair completed: port=0, LBA=0x2560A19<br />
c0 [Sat Aug 25 03:43:58 2012] WARNING Sector repair completed: port=0, LBA=0x2560A1B<br />
c0 [Sat Aug 25 03:44:30 2012] WARNING Sector repair completed: port=0, LBA=0x2560A21<br />
c0 [Sat Aug 25 20:58:14 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 29 04:57:15 2012] WARNING Sector repair completed: port=1, LBA=0xF3957EB<br />
c0 [Sat Sep 1 00:01:15 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 1 03:21:52 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC86<br />
c0 [Sat Sep 1 03:22:15 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC88<br />
c0 [Sat Sep 1 03:22:34 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC8A<br />
c0 [Sat Sep 1 03:22:47 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC8C<br />
c0 [Sat Sep 1 17:17:22 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD1<br />
c0 [Sat Sep 1 17:17:45 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD3<br />
c0 [Sat Sep 1 17:18:04 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD5<br />
c0 [Sat Sep 1 21:36:56 2012] INFO Verify completed: unit=0<br />
c0 [Thu Sep 6 00:07:30 2012] WARNING Sector repair completed: port=0, LBA=0xDA3C64B<br />
c0 [Thu Sep 6 00:32:56 2012] WARNING Sector repair completed: port=1, LBA=0x6BBA816<br />
c0 [Sat Sep 8 00:01:13 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 8 00:09:56 2012] WARNING Sector repair completed: port=0, LBA=0xDEBC958<br />
c0 [Sat Sep 8 04:38:45 2012] WARNING Sector repair completed: port=0, LBA=0x38D254F<br />
c0 [Sat Sep 8 20:44:50 2012] INFO Verify completed: unit=0<br />
c0 [Mon Sep 10 01:26:34 2012] WARNING Sector repair completed: port=1, LBA=0xFFD8D5E<br />
c0 [Wed Sep 12 00:33:48 2012] WARNING Sector repair completed: port=1, LBA=0xE8DB928<br />
c0 [Wed Sep 12 00:36:33 2012] WARNING Sector repair completed: port=1, LBA=0x6D49411<br />
c0 [Fri Sep 14 01:59:39 2012] WARNING Sector repair completed: port=0, LBA=0x1467F1C<br />
c0 [Fri Sep 14 02:08:27 2012] WARNING Sector repair completed: port=1, LBA=0x14C8ABD<br />
c0 [Fri Sep 14 03:54:47 2012] WARNING Sector repair completed: port=0, LBA=0x1580C915<br />
c0 [Sat Sep 15 00:01:11 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 15 02:38:14 2012] WARNING Sector repair completed: port=0, LBA=0x1C178973<br />
c0 [Sat Sep 15 02:59:02 2012] WARNING Sector repair completed: port=0, LBA=0x1C178975<br />
c0 [Sat Sep 15 04:47:08 2012] WARNING Sector repair completed: port=0, LBA=0x3FA0356<br />
c0 [Sat Sep 15 04:47:31 2012] WARNING Sector repair completed: port=0, LBA=0x3FA0359<br />
c0 [Sat Sep 15 10:41:59 2012] WARNING Sector repair completed: port=0, LBA=0x6DFD1EC<br />
c0 [Sat Sep 15 13:25:23 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD100<br />
c0 [Sat Sep 15 13:25:31 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD104<br />
c0 [Sat Sep 15 13:25:54 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD106<br />
c0 [Sat Sep 15 17:10:50 2012] WARNING Sector repair completed: port=0, LBA=0x1C178977<br />
c0 [Sat Sep 15 20:59:57 2012] INFO Verify completed: unit=0<br />
c0 [Tue Sep 18 01:17:18 2012] WARNING Sector repair completed: port=1, LBA=0x803B05B<br />
c0 [Sat Sep 22 00:01:10 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 22 20:54:31 2012] INFO Verify completed: unit=0<br />
c0 [Tue Sep 25 01:56:47 2012] WARNING Sector repair completed: port=0, LBA=0x26E3909<br />
c0 [Sat Sep 29 00:01:08 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 29 02:04:14 2012] WARNING Sector repair completed: port=0, LBA=0x146AC03<br />
c0 [Sat Sep 29 10:58:39 2012] WARNING Sector repair completed: port=0, LBA=0x6D4EB0E<br />
c0 [Sat Sep 29 10:59:02 2012] WARNING Sector repair completed: port=0, LBA=0x6D4EB14<br />
c0 [Sat Sep 29 11:22:44 2012] WARNING Sector repair completed: port=0, LBA=0x6F79623<br />
c0 [Sat Sep 29 13:50:48 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D65E<br />
c0 [Sat Sep 29 13:51:11 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D661<br />
c0 [Sat Sep 29 13:51:30 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D663<br />
c0 [Sat Sep 29 20:57:34 2012] INFO Verify completed: unit=0<br />
c0 [Mon Oct 1 04:47:24 2012] WARNING Sector repair completed: port=0, LBA=0xC5BC6F2<br />
c0 [Tue Oct 2 02:00:27 2012] WARNING Sector repair completed: port=0, LBA=0x1547667<br />
c0 [Tue Oct 2 02:01:56 2012] WARNING Sector repair completed: port=0, LBA=0x154766F<br />
c0 [Tue Oct 2 05:02:31 2012] WARNING Sector repair completed: port=1, LBA=0xD67D054<br />
c0 [Tue Oct 2 05:04:14 2012] WARNING Sector repair completed: port=1, LBA=0xD67D056<br />
c0 [Wed Oct 3 01:22:12 2012] WARNING Sector repair completed: port=1, LBA=0x12AAF8CA<br />
c0 [Thu Oct 4 04:29:22 2012] WARNING Sector repair completed: port=0, LBA=0x13E6F992<br />
c0 [Thu Oct 4 05:10:51 2012] WARNING Sector repair completed: port=0, LBA=0x1C252A4<br />
c0 [Sat Oct 6 00:01:07 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 6 19:41:18 2012] WARNING Sector repair completed: port=1, LBA=0x5A5C3AE8<br />
c0 [Sat Oct 6 21:01:05 2012] INFO Verify completed: unit=0<br />
c0 [Mon Oct 8 00:32:06 2012] WARNING Sector repair completed: port=0, LBA=0x6C60D3E<br />
c0 [Tue Oct 9 03:51:03 2012] WARNING Sector repair completed: port=1, LBA=0x89B5EC9<br />
c0 [Thu Oct 11 04:21:17 2012] WARNING Sector repair completed: port=1, LBA=0x13F85833<br />
c0 [Sat Oct 13 00:01:05 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 13 05:12:40 2012] WARNING Sector repair completed: port=0, LBA=0x3FA5134<br />
c0 [Sat Oct 13 21:08:35 2012] INFO Verify completed: unit=0<br />
c0 [Tue Oct 16 03:53:50 2012] WARNING Sector repair completed: port=1, LBA=0x148AA1BD<br />
c0 [Thu Oct 18 03:20:30 2012] WARNING Sector repair completed: port=0, LBA=0x1C8DABCB<br />
c0 [Thu Oct 18 04:52:50 2012] WARNING Sector repair completed: port=0, LBA=0xE879057<br />
c0 [Sat Oct 20 00:01:04 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 20 02:19:25 2012] WARNING Sector repair completed: port=1, LBA=0x174B012<br />
c0 [Sat Oct 20 03:41:38 2012] WARNING Sector repair completed: port=0, LBA=0x256D93B<br />
c0 [Sat Oct 20 03:42:01 2012] WARNING Sector repair completed: port=0, LBA=0x256D93D<br />
c0 [Sat Oct 20 03:42:40 2012] WARNING Sector repair completed: port=0, LBA=0x256D940<br />
c0 [Sat Oct 20 03:42:59 2012] WARNING Sector repair completed: port=0, LBA=0x256D942<br />
c0 [Sat Oct 20 03:43:12 2012] WARNING Sector repair completed: port=0, LBA=0x256D944<br />
c0 [Sat Oct 20 03:43:26 2012] WARNING Sector repair completed: port=0, LBA=0x256D948<br />
c0 [Sat Oct 20 16:37:52 2012] WARNING Sector repair completed: port=0, LBA=0xE879060<br />
c0 [Sat Oct 20 16:38:15 2012] WARNING Sector repair completed: port=0, LBA=0xE879062<br />
c0 [Sat Oct 20 21:00:18 2012] INFO Verify completed: unit=0<br />
c0 [Sat Oct 20 23:49:01 2012] WARNING Sector repair completed: port=1, LBA=0x4473E908<br />
c0 [Sun Oct 21 03:42:26 2012] WARNING Sector repair completed: port=1, LBA=0x175BADD5<br />
c0 [Tue Oct 23 01:09:04 2012] WARNING Sector repair completed: port=1, LBA=0x6E524860<br />
c0 [Fri Oct 26 03:21:25 2012] WARNING Sector repair completed: port=0, LBA=0x802C61<br />
c0 [Fri Oct 26 04:22:21 2012] WARNING Sector repair completed: port=0, LBA=0x176353CD<br />
c0 [Sat Oct 27 00:01:03 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 27 00:49:35 2012] WARNING Sector repair completed: port=0, LBA=0x802C65<br />
c0 [Sat Oct 27 17:02:24 2012] WARNING Sector repair completed: port=1, LBA=0xC1FF26D<br />
c0 [Sat Oct 27 17:09:06 2012] WARNING Sector repair completed: port=0, LBA=0xDF621AD<br />
c0 [Sat Oct 27 21:30:57 2012] INFO Verify completed: unit=0<br />
c0 [Tue Oct 30 00:20:46 2012] WARNING Sector repair completed: port=0, LBA=0xE9FE2AB<br />
c0 [Wed Oct 31 02:02:03 2012] WARNING Sector repair completed: port=0, LBA=0x1460C25<br />
c0 [Wed Oct 31 02:04:05 2012] WARNING Sector repair completed: port=0, LBA=0x1460C28<br />
c0 [Thu Nov 1 00:48:34 2012] WARNING Sector repair completed: port=1, LBA=0xA7C92BE<br />
c0 [Thu Nov 1 05:04:45 2012] WARNING Sector repair completed: port=0, LBA=0x1C252C2<br />
<br />
[root@backup1 /data/deprecated]#<br />
</pre><br />
<br />
Look for failed drives and errors. Obviously from the above we need to probably replace drives 0 and 1 and drive 1 is even showing as having problems, yet the RAID array is healthy, amazingly. You also see the automatic verifies.<br />
<br />
Note: when rebuilding a degraded mirror, you will see no progress as it rebuilds in the cli<br />
<pre>Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u0 RAID-5 REBUILDING 0 - 64K 4656.56 OFF ON</pre><br />
<br />
<br />
=== areca ===<br />
<br />
We are using an areca controller on backup3. <br />
<br />
<pre><br />
[root@newbackup3 ~]# sh /root/verify.sh<br />
# Name Raid Name Level Capacity Ch/Id/Lun State<br />
===============================================================================<br />
1 ARC-1160-VOL#00 Raid Set # 00 Raid5 5000.0GB 00/00/00 Checking(19.7%)<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
# Name Disks TotalCap FreeCap DiskChannels State<br />
===============================================================================<br />
1 Raid Set # 00 6 6000.0GB 0.0GB 123456 Checking<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
Date-Time Device Event Type Elapsed Time Errors<br />
===============================================================================<br />
2012-12-05 20:40:58 ARC-1160-VOL#00 Start Checking<br />
2012-12-01 05:06:04 ARC-1160-VOL#00 Complete Init 027:30:45<br />
2012-11-30 01:35:19 ARC-1160-VOL#00 Start Initialize<br />
2026-08-06 01:34:52 H/W MONITOR Raid Powered On<br />
2012-11-30 01:33:36 ARC-1160-VOL#00 Stop Initialization 000:31:48<br />
2012-11-30 01:01:47 ARC-1160-VOL#00 Start Initialize<br />
2026-08-06 00:58:13 H/W MONITOR Raid Powered On<br />
2012-11-30 00:57:26 ARC-1160-VOL#00 Stop Initialization 000:57:07<br />
2012-11-30 00:00:19 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 23:56:48 H/W MONITOR Raid Powered On<br />
2026-08-05 23:52:58 H/W MONITOR Raid Powered On<br />
2026-08-05 23:50:14 H/W MONITOR Raid Powered On<br />
2026-08-05 23:43:30 H/W MONITOR Raid Powered On<br />
2012-11-29 23:10:07 ARC-1160-VOL#00 Stop Initialization 000:00:56<br />
2012-11-29 23:09:11 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 23:08:57 H/W MONITOR Raid Powered On<br />
2012-11-29 23:08:10 ARC-1160-VOL#00 Stop Initialization 000:20:41<br />
2012-11-29 22:47:29 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 22:46:59 H/W MONITOR Raid Powered On<br />
2026-08-05 22:45:55 H/W MONITOR Raid Powered On<br />
2026-08-05 22:44:53 H/W MONITOR Raid Powered On<br />
2026-08-05 22:42:06 H/W MONITOR Raid Powered On<br />
2026-08-05 22:40:50 H/W MONITOR Raid Powered On<br />
2012-11-29 22:40:04 ARC-1160-VOL#00 Stop Initialization 000:24:25<br />
2012-11-29 22:15:38 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 22:15:11 000:000001215B00 Restart Init LBA Point<br />
2026-08-05 22:15:10 H/W MONITOR Raid Powered On<br />
2012-11-29 21:56:38 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 21:56:12 H/W MONITOR Raid Powered On<br />
2026-08-05 21:56:04 IDE Channel #03 Device Inserted<br />
2012-11-29 21:55:13 IDE Channel #04 Device Inserted<br />
2012-11-29 21:55:03 IDE Channel #02 Device Inserted<br />
2026-08-05 21:53:09 H/W MONITOR Raid Powered On<br />
2026-08-05 20:51:46 H/W MONITOR Raid Powered On<br />
2026-08-05 20:49:56 H/W MONITOR Raid Powered On<br />
2026-08-05 20:48:29 H/W MONITOR Raid Powered On<br />
2026-08-05 20:46:29 H/W MONITOR Raid Powered On<br />
2026-08-05 20:44:49 H/W MONITOR Raid Powered On<br />
2026-08-05 20:43:01 H/W MONITOR Raid Powered On<br />
2026-08-05 20:36:25 H/W MONITOR Raid Powered On<br />
2026-08-05 20:31:18 H/W MONITOR Raid Powered On<br />
2026-08-05 20:30:08 H/W MONITOR Raid Powered On<br />
2026-08-05 20:08:40 H/W MONITOR Raid Powered On<br />
2026-08-05 20:06:11 H/W MONITOR Raid Powered On<br />
2026-08-05 20:05:14 H/W MONITOR Raid Powered On<br />
2026-08-05 20:03:58 H/W MONITOR Raid Powered On<br />
2026-08-05 20:00:56 H/W MONITOR Raid Powered On<br />
2026-08-05 19:57:57 H/W MONITOR Raid Powered On<br />
2026-08-05 19:56:15 H/W MONITOR Raid Powered On<br />
2026-08-05 19:55:05 H/W MONITOR Raid Powered On<br />
2026-08-05 17:24:36 H/W MONITOR Raid Powered On<br />
2026-08-05 17:22:43 H/W MONITOR Raid Powered On<br />
2026-08-05 04:50:42 H/W MONITOR Raid Powered On<br />
2026-08-05 04:47:33 H/W MONITOR Raid Powered On<br />
2026-08-05 04:43:57 H/W MONITOR Raid Powered On<br />
2026-08-05 04:18:52 H/W MONITOR Raid Powered On<br />
2026-08-05 04:17:30 H/W MONITOR Raid Powered On<br />
2026-08-05 04:13:30 H/W MONITOR Raid Powered On<br />
2026-08-05 04:10:26 H/W MONITOR Raid Powered On<br />
2026-08-05 04:09:23 H/W MONITOR Raid Powered On<br />
2026-08-05 00:08:09 H/W MONITOR Raid Powered On<br />
2026-08-05 00:07:12 H/W MONITOR Raid Powered On<br />
2026-08-05 00:05:51 H/W MONITOR Raid Powered On<br />
2026-08-05 00:04:27 H/W MONITOR Raid Powered On<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
press enter when ready to run verify<br />
</pre><br />
<br />
Look for failed drives and errors. <br />
<br />
When it proceed's to verifying, you can confirm with:<br />
<br />
<pre><br />
[root@newbackup3 ~]# cli64 vsf info<br />
# Name Raid Name Level Capacity Ch/Id/Lun State<br />
===============================================================================<br />
1 ARC-1160-VOL#00 Raid Set # 00 Raid5 5000.0GB 00/00/00 Checking(22.5%)<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
[root@newbackup3 ~]#<br />
</pre><br />
<br />
== Update OS list ==<br />
<br />
# check for any new VZ templates we want to offer: <tt>vzup2date -z</tt><br />
# see if there's any OS's we want to include in our colo install list. Update 2 places: <tt>signup/html/colo_quote.html</tt> & <tt>signup/html/step1.html</tt><br />
# update the mgmt database (ref_templates table, ref_systems table).<br />
<br />
= Infrequent tasks =<br />
<br />
== Free up space on gateway ==<br />
<br />
<pre>newgateway /var/spool# cd clientmqueue/<br />
newgateway /var/spool/clientmqueue# sh<br />
# for f in `ls`; do rm $f; done<br />
exit</pre><br />
<br />
== Free up space on mail ==<br />
<br />
You can clear out root mail:<br />
<pre><br />
mail /var/log# ll -h /var/mail/root<br />
-rw------- 1 root mail 543K Dec 19 13:05 /var/mail/root<br />
mail /var/log# rm /var/mail/root<br />
</pre><br />
<br />
Or you can archive mail logs:<br />
<pre><br />
mail /var/log# ls -l htt*<br />
-rw-r--r-- 1 root wheel 297436931 Dec 19 13:26 httpd-access.log<br />
-rw-r--r-- 1 root wheel 9824324 Jul 4 11:34 httpd-access.log.old.0.gz<br />
-rw-r--r-- 1 root wheel 6884137 Mar 17 2012 httpd-access.log.old.1.gz<br />
-rw-r--r-- 1 root wheel 18557444 Dec 3 2009 httpd-access.log.old.10.gz<br />
-rw-r--r-- 1 root wheel 14740263 Jan 9 2007 httpd-access.log.old.11.gz<br />
-rw-r--r-- 1 root wheel 14209465 Nov 28 2007 httpd-access.log.old.12.gz<br />
-rw-r--r-- 1 root wheel 16874396 Feb 19 2012 httpd-access.log.old.3.gz<br />
-rw-r--r-- 1 root wheel 14554859 Jul 22 2011 httpd-access.log.old.4.gz<br />
-rw-r--r-- 1 root wheel 10513227 Feb 18 2011 httpd-access.log.old.5.gz<br />
-rw-r--r-- 1 root wheel 7201946 Oct 29 2010 httpd-access.log.old.6.gz<br />
-rw-r--r-- 1 root wheel 10062537 May 6 2010 httpd-access.log.old.7.gz<br />
-rw-r--r-- 1 root wheel 10157042 Aug 12 2010 httpd-access.log.old.8.gz<br />
-rw-r--r-- 1 root wheel 11909534 Mar 4 2010 httpd-access.log.old.9.gz<br />
-rw-r--r-- 1 root wheel 59030930 Dec 19 13:01 httpd-error.log<br />
-rw-r--r-- 1 root wheel 3413134 Mar 4 2010 httpd-error.log.0.gz<br />
-rw-r--r-- 1 root wheel 795515 May 1 2007 httpd-error.log.1.gz<br />
-rw-r--r-- 1 root wheel 1142153 Nov 30 2007 httpd-error.log.2.gz<br />
-rw-r--r-- 1 root wheel 2325801 Feb 18 2011 httpd-error.log.gz<br />
<br />
mail /var/log# sh<br />
# for f in 12 11 10 9 8 7 6 5 4 3 2 1 0; do g=`echo $f+1|bc`; mv httpd-access.log.old.$f.gz httpd-access.log.old.$g.gz; done<br />
# mv httpd-access.log httpd-access.log.old.0<br />
# touch httpd-access.log<br />
# apachectl restart<br />
# gzip httpd-access.log.old.0<br />
<br />
# for f in 2 1 0; do g=`echo $f+1|bc`; mv httpd-error.log.$f.gz httpd-error.log.$g.gz; done<br />
# mv httpd-error.log httpd-error.log.0<br />
# touch httpd-error.log<br />
# apachectl restart<br />
# gzip httpd-error.log.0<br />
# exit<br />
</pre><br />
<br />
== Free up space on bwdb2 ==<br />
<br />
You can either remove items from <tt>/usr/home/archive</tt> or you can scp them to backup3:/data/bwdb2/archive .<br />
<br />
== Free up space on backup1 ==<br />
<br />
backup1 is our primary customer backup system. As usage grows over time, it needs to be regularly purged of old files. The easiest way to do this is by removing deprecated files. These mostly consist of cancelled customers or temporary dump/storage files (created during dump/restores). Our standard policy is to hang onto cancelled customers for 6mos after which we remove their files (as far as customers know their data is purged immediately, but we hang onto it just in case.. and in some cases we cancel a server due to non payment so this makes it easy to restore their system). To find files to remove:<br />
<br />
<pre>[root@backup1 ~]# cd /data/deprecated/<br />
[root@backup1 /data/deprecated]# ls<br />
2101-migrated-20120317.tgz old-683-cxld-20121021.tgz<br />
69.55.230.2-wwwbackup old-744-cxld-20120708.tgz<br />
991-DONTDELETE.tgz old-809-cxld-20120609.tgz<br />
archive-col02050-mdfile-cxld-20120409.gz old-854-cxld-20120621.tgz<br />
col01371.tgz old-931-cxld-20060513.tgz<br />
deleteme_ubuntu-10.10-x86_20111205 old-col00123-mdfile-noarchive-20120417.gz<br />
jail10_old old-col00147-vnfile-cxld-20120828.gz<br />
jail14_rsync_old old-col00419-dump-cxld-20120224.gz<br />
jail15_old old-col01098-vnfile-cxld-20120827.gz<br />
jail3_old old-col01278-dump-cxld-20120822<br />
jail4_old old-col01517-dump-cxld-20120828<br />
jail5_old old-col01669-dump-cxld-20120203.gz<br />
old-1009-cxld-20120608.tgz old-col01687-dump-cxld-20120909<br />
old-1012-cxld-20120411.tgz old-col01790-dump-cxld-20120828<br />
old-1052-cxld-20120721.tgz old-col01812-dump-cxld-20120820<br />
old-10631-cxld-20120622.tgz old-col01938-mdfile-cxld-20120619.gz<br />
old-10632-cxld-20120622.tgz old-col02095-mdfile-noarchive-20120523.gz<br />
old-10633-cxld-20120622.tgz olddebian-3.0-v15-20110610.tgz<br />
old-1236-cxld-20120621.tgz oldmod_frontpage-deb30-v15-20110610.tgz<br />
old-1381-cxld-20120404.tgz oldmod_perl-deb30-v15-20110610.tgz<br />
old-1422-cxld-20120721.tgz oldmod_ssl-deb30-v15-20110610.tgz<br />
old-14681-cxld-20120619.tgz oldmysql-deb30-v15-20110610.tgz<br />
old-1544-cxld-20120626.tgz oldproftpd-deb30-v15-20110610.tgz<br />
old-18351-cxld-20120605.tgz old_virt14<br />
old-1853-cxld-20120910.tgz old_virt18<br />
old-1963-cxld-20120206.tgz oldwebmin-deb30-v15-20110610.tgz<br />
old-1967-cxld-20120605.tgz suse.virt11.20120421.tgz<br />
old-1981-noarchive-20120729.tgz virt11<br />
old-2030-migrated-noarchive-20120727.tgz virt12_old<br />
old-2037-cxld-20120716.tgz virt13_old<br />
old-2065-cxld-20120727.tgz virt16_old<br />
old-2068-cxld-20120424.tgz virt4_old<br />
old-2085-cxld-20120531.tgz virt5_old<br />
old-364-cxld-20120904.tgz virt6_old<br />
old-446-cxld-20120512.tgz virt7_old<br />
old-613-cxld-20120601.tgz virt8_old<br />
[root@backup1 /data/deprecated]#<br />
</pre><br />
<br />
virtX_old and jailX_old are permanently archived, so ignore those as well as anything else marked not to delete or otherwise suspicious. Likewise, probably a good idea to try to hang onto oldTEMPLATE.gz as long as we can as well.<br />
Most of the stuff we want to delete is dated when it was deprecated, making this easy. So to remove files from 6 mos ago (running this in Oct):<br />
<br />
<pre>[root@backup1 /data/deprecated]# ls old*201204*<br />
old-1012-cxld-20120411.tgz old-2068-cxld-20120424.tgz<br />
old-1381-cxld-20120404.tgz old-col00123-mdfile-noarchive-20120417.gz<br />
[root@backup1 /data/deprecated]# rm old*201204*<br />
</pre><br />
<br />
Every few months you will also want to remove some of the snapshot archives for mail. We typically save the 1st, 10th, and 20th of each month. To do this you set aside the dates you want to save then remove months at a time, followed by restoring the set aside dates. Here's how that works:<br />
<pre>[root@backup1 /data/www/daily]# ls<br />
05 08-10-11 10-04-10 11-10-10 12-07-29 12-09-21 12-11-14<br />
06 08-10-21 10-04-20 11-10-20 12-07-30 12-09-22 12-11-15<br />
06-06-01-usr-home.tgz 08-11-01 10-05-01 11-11-01 12-07-31 12-09-23 12-11-16<br />
06-07-01-usr-home.tgz 08-11-10 10-05-11 11-11-10 12-08-01 12-09-24 12-11-17<br />
06-08-01-usr-home.tgz 08-11-20 10-05-20 11-11-20 12-08-02 12-09-25 12-11-18<br />
06-09-01-usr-home.tgz 08-12-01 10-06-01 11-12-01 12-08-03 12-09-26 12-11-19<br />
06-11-10 08-12-10 10-06-10 11-12-10 12-08-04 12-09-27 12-11-20<br />
06-12-21 08-12-20 10-06-20 11-12-20 12-08-05 12-09-28 12-11-21<br />
07-01-10 09-01-01 10-07-01 12-01-01 12-08-06 12-09-29 12-11-22<br />
07-01-20 09-01-10 10-07-10 12-01-10 12-08-07 12-09-30 12-11-23<br />
07-02-10 09-01-20 10-07-20 12-01-20 12-08-08 12-10-01 12-11-24<br />
07-02-20 09-02-01 10-08-01 12-02-01 12-08-09 12-10-02 12-11-25<br />
07-03-01 09-02-10 10-08-10 12-02-10 12-08-10 12-10-03 12-11-26<br />
07-03-20 09-02-20 10-08-20 12-02-20 12-08-11 12-10-04 12-11-27<br />
07-04-01 09-03-01 10-09-01 12-03-01 12-08-12 12-10-05 12-11-28<br />
07-04-10 09-03-10 10-09-10 12-03-10 12-08-13 12-10-06 12-11-29<br />
07-04-20 09-03-20 10-09-20 12-03-20 12-08-14 12-10-07 12-11-30<br />
07-05-01 09-04-01 10-10-01 12-04-01 12-08-15 12-10-08 12-12-01<br />
07-05-10 09-04-10 10-10-10 12-04-10 12-08-16 12-10-09 12-12-02<br />
07-05-20 09-04-20 10-10-20 12-04-20 12-08-17 12-10-10 12-12-03<br />
07-06-01 09-05-01 10-11-01 12-05-01 12-08-18 12-10-11 12-12-04<br />
07-06-10 09-05-10 10-11-10 12-05-10 12-08-19 12-10-12 12-12-05<br />
07-06-20 09-05-20 10-11-20 12-05-20 12-08-20 12-10-13 12-12-06<br />
07-07-20 09-06-01 10-12-01 12-06-01 12-08-21 12-10-14 12-12-07<br />
07-08-10 09-06-10 10-12-10 12-06-10 12-08-22 12-10-15 12-12-08<br />
07-08-20 09-06-20 10-12-20 12-06-20 12-08-23 12-10-16 12-12-09<br />
07-09-01 09-07-01 11-01-01 12-07-01 12-08-24 12-10-17 12-12-10<br />
07-10-01 09-07-10 11-01-10 12-07-02 12-08-25 12-10-18 12-12-11<br />
07-10-10 09-07-20 11-01-21 12-07-03 12-08-26 12-10-19 12-12-12<br />
07-10-20 09-08-01 11-02-01 12-07-04 12-08-27 12-10-20 12-12-13<br />
07-12-01 09-08-10 11-02-10 12-07-05 12-08-28 12-10-21 12-12-14<br />
07-12-10 09-08-20 11-02-20 12-07-06 12-08-29 12-10-22 12-12-15<br />
08-01-01 09-09-01 11-03-01 12-07-07 12-08-30 12-10-23 12-12-16<br />
08-01-20 09-09-10 11-03-10 12-07-08 12-08-31 12-10-24 12-12-17<br />
08-02-20 09-09-20 11-03-20 12-07-09 12-09-01 12-10-25 12-12-18<br />
08-03-01 09-10-01 11-04-01 12-07-10 12-09-02 12-10-26 12-12-19<br />
08-03-10 09-10-10 11-04-10 12-07-11 12-09-03 12-10-27 12-12-20<br />
08-03-20 09-10-20 11-04-20 12-07-12 12-09-04 12-10-28 12-12-21<br />
08-04-01 09-11-01 11-05-01 12-07-13 12-09-05 12-10-29 12-12-22<br />
08-04-20 09-11-10 11-05-10 12-07-14 12-09-06 12-10-30 12-12-23<br />
08-05-01 09-11-20 11-05-20 12-07-15 12-09-07 12-10-31 12-12-24<br />
08-05-10 09-12-01 11-06-01 12-07-16 12-09-08 12-11-01 12-12-25<br />
08-06-10 09-12-10 11-06-10 12-07-17 12-09-09 12-11-02 12-12-26<br />
08-06-20 09-12-20 11-06-20 12-07-18 12-09-10 12-11-03 12-12-27<br />
08-07-02 10-01-01 11-07-01 12-07-19 12-09-11 12-11-04 12-12-28<br />
08-07-10 10-01-10 11-07-10 12-07-20 12-09-12 12-11-05 2008-10-23<br />
08-07-20 10-01-20 11-07-20 12-07-21 12-09-13 12-11-06 bb.tgz<br />
08-08-01 10-02-01 11-08-01 12-07-22 12-09-14 12-11-07 boot<br />
08-08-10 10-02-10 11-08-10 12-07-23 12-09-15 12-11-08 current<br />
08-08-21 10-02-20 11-08-20 12-07-24 12-09-16 12-11-09 hold<br />
08-09-01 10-03-01 11-09-01 12-07-25 12-09-17 12-11-10<br />
08-09-10 10-03-10 11-09-10 12-07-26 12-09-18 12-11-11<br />
08-09-21 10-03-20 11-09-20 12-07-27 12-09-19 12-11-12<br />
08-10-01 10-04-01 11-10-01 12-07-28 12-09-20 12-11-13<br />
[root@backup1 /data/www/daily]#<br />
</pre><br />
<br />
So we see that everything up to July 2012 has been pruned. To prune July 2012 we do the following:<br />
<br />
<pre>mv 12-07-01 hold<br />
mv 12-07-10 hold<br />
mv 12-07-20 hold<br />
rm -fr 12-07*<br />
mv hold/* .</pre></div>75.80.128.147https://wiki.jcihosting.com/index.php?title=Routine_Maintenance&diff=2203Routine Maintenance2020-02-25T17:57:50Z<p>75.80.128.147: /* Outgoing */</p>
<hr />
<div>= Daily Tasks =<br />
<br />
== check load graphs == <br />
<br />
Click on the Load link in mgmt <br />
<br />
This screen shows you load levels on our servers and network traffic for critical machines (firewalls, backup servers).<br />
<br />
If you see load high or increasing <br />
<br />
FreeBSD: <br />
run [[VPS_Management#jtop|jtop]] (or [[VPS_Management#jt|jt]] > 7.x) and see if there are any runaway processes. Here are some examples of entries in top that are<br />
definitely runaway processes:<br />
<br />
<pre>79481 root 64 0 2256K 1056K CPU1 1 58:16 87.40% 87.40% nano<br />
50650 1000 64 0 1852K 1112K RUN 0 207.9H 84.08% 84.08% screen<br />
14829 www 2 0 39100K 31736K accept 0 104:24 46.54% 6.54% httpd<br />
42065 root 61 0 1300K 844K RUN 1 47.8H 91.36% 91.36% ee<br />
1328 www 56 0 18440K 10796K CPU1 0 64.4H 97.71% 97.71% httpd<br />
26251 user 57 0 6124K 1160K CPU1 1 82.9H 98.44% 98.44% screen<br />
89874 root 60 0 1352K 892K RUN 1 33.8H 65.82% 65.82% dialog<br />
38656 1000 64 0 3088K 2136K CPU0 0 806:13 97.95% 97.95% StutBot<br />
27630 root 64 0 1396K 972K RUN 1 76.8H 86.47% 86.47% ee</pre><br />
<br />
Linux:<br />
run [[VPS_Management#vwe|vwe]] to see which VPS’s have high loads. From there run <tt>[[VPS_Management#vp|vp]] <veid></tt> and/or <tt>[[VPS_Management#vt|vt]] <veid></tt> to see what's going on in that system.<br />
[[VPS_Management#vzstat|vzstat]] will also give you a nice picture of whats going on, systems with high numbers in the mlat column are likely culprits.<br />
<br />
examples of out of control procs:<br />
<br />
<pre>12183 nobody 16 0 4916 1348 1340 R 45.5 0.0 4249m httpd<br />
29266 #502 16 0 1852 796 792 R 22.5 0.0 1104m vim<br />
23860 #41 16 0 5472 5472 2076 R 98.9 0.2 31:41 python<br />
19227 bin 19 0 1688 716 652 R 99.9 0.0 321:08 wtrs_ui<br />
7762 apache 16 0 268 236 224 R 85.7 0.0 1010m ptrace<br />
4624 #501 20 0 4304 2400 2044 R 53.6 0.1 284:32 YoSucker<br />
20451 #506 20 0 1876 820 816 R 17.2 0.0 169:35 vim<br />
8834 #514 20 0 900 724 672 R 77.6 0.0 382:30 neostats<br />
31815 apache 14 0 3176 3176 1696 R 74.4 0.1 6:15 counter</pre><br />
<br />
Just kill -9 them and be done with it.<br />
<br />
Also, anytime you see `kmod` or `ptrace` - kill those immediaely no matter how much they are using - they are attempts to exploit the linux ptrace bug. They won't work, but they suck a lot of CPU...<br />
<br />
Also, any other processes that are at 90-100% cpu usage and have been running for any long period of time should be killed except for mysqld processes on FreeBSD. See above.<br />
<br />
However, there is an exception:<br />
<br />
if it is a mysqld, we don't want to kill their database. What you want to do is <tt>[[VPS_Management#jpid|jpid]] <pid></tt> to see who owns it, and then email them the paste containing the instructions for the nanny. Or you can simply do a <tt>kill -1 PID</tt> on the process to restart it.<br />
<br />
=== Load averages jump at night ===<br />
<br />
The load averages on the FreeBSD systems may jump up at night between 1 and 4 am - this is because the backups are running - if this is what is causing a jump in load, you will see processes like `rsync` in top eating a lot of CPU time.<br />
<br />
== check backups ==<br />
<br />
mgmt -> Motnroing -> Backups and make sure every machine was backed up the previous nite. <br />
Also look at df on backup1 and backup2 to make sure no disk is approaching full, though bb should warn us in advance. Please note - errors encountered when a backup script on any of the particular systems run will gnerate an email to support@johncompanies.com so you can know immediately the day after if the directory to be backed up has been moved or no longer exists. A paste exists for this to notify the customer of a non-existant file/dir.<br />
<br />
== check bb for warnings ==<br />
mgmt -> BigBrother<br />
<br />
Some events don't generate pages (on purpose). You will only see them by going to the bb main page.<br />
<br />
== check jail5 for crashed VPSs ==<br />
<br />
On jail5<br />
<br />
notrunning<br />
<br />
To restart a VPS <br />
<br />
vm restart col0XXXX<br />
<br />
== check NetHere ==<br />
<br />
Check the NetHere servers<br />
<br />
=== Mail systems ===<br />
Check for possible SPAMMERs.<br />
<br />
==== Incoming ====<br />
Check mta-1 and mta-2 count of customer logins for possible<br />
customer SPAM compromises.<br />
<br />
login_count /logs/maillog | tail -30<br />
<br />
==== Outgoing ====<br />
<br />
Check outgoing queues on relay-1 and relay-2<br />
<br />
mail_count | tail -30<br />
<br />
To clean up outgoing queues of unwanted SPAM.<br />
<br />
mail_cleanup [<sender's domain>]<br />
<br />
=== Nagios ===<br />
<br />
Check for other problems on NetHere servers<br />
<br />
https://nagios.nethere.net<br />
<br />
=== Cacti ===<br />
<br />
Check bandwidth usage on servers<br />
<br />
https://cacti.nethere.net<br />
<br />
= Monthly Tasks =<br />
== rotate pine sent mail (1st of month) ==<br />
On the 1st of the month, before any emails are sent out, quit out of pine, then log back in. Send mail from last month will be archived.<br />
If you mess up and do it on the 3rd (for example), you can go into the previous month's saved email and save emails from the current month into the <tt>sent-mail</tt> (current month) mailbox.<br />
<br />
== b/w caps ==<br />
On the 1st: remove any bwcaps put into the firewall (only really applies if a bwcap was added cause someone went over on b/w):<br />
<br />
ipfw list|grep pipe<br />
ipfw del [each rule listed]<br />
<br />
NOTE: this cronjob on newgateway will do some of that for you, provided you used one of the following pipe #s:<br />
0 0 1 * * /sbin/ipfw del 3 4 5 17331<br />
<br />
-------------------<br />
<br />
We really don’t do this anymore since we have centralized traffic accounting with netflow, but for posterity:<br />
<br />
Make sure all machines reset counters to 0 after midnight on the 1st<br />
Make sure they dumped a counter<br />
<br />
On each jail run:<br />
trafficgather.pl<br />
<br />
And on each virt:<br />
linuxtrafficgather.pl<br />
<br />
== Monthly RAID checks ==<br />
<br />
Every month we check the health of and verfy the parity on all our RAID-based systems.<br />
To facilitate this, we've created a simple script to start the process:<br />
<br />
sh /root/verify.sh<br />
<br />
=== Adaptec controllers ===<br />
Here's some sample output:<br />
<pre><br />
mail /usr/local/www/scripts# sh /root/verify.sh<br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> container list /f<br />
Executing: container list /full=TRUE<br />
Num Total Oth Chunk Scsi Partition<br />
Creation System<br />
Label Type Size Ctr Size Usage B:ID:L Offset:Size State RO Lk Task Done% Ent<br />
Date Time Files<br />
----- ------ ------ --- ------ ------- ------ ------------- ------- -- -- ------- ------ ---<br />
------ -------- ------<br />
0 Mirror 33.9GB Open 0:01:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:32<br />
/dev/aacd0 mirror0 0:00:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:32<br />
<br />
1 Mirror 33.9GB Open 0:02:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:50<br />
/dev/aacd1 mirror1 0:03:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:50<br />
<br />
<br />
AAC0> disk list /f<br />
Executing: disk list /full=TRUE<br />
<br />
B:ID:L Device Type Removable media Vendor-ID Product-ID Rev Blocks Bytes/Bl<br />
ock Usage Shared Rate<br />
------ -------------- --------------- --------- ---------------- ----- --------- --------<br />
--- ---------------- ------ ----<br />
0:00:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:01:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:02:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:03:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
<br />
AAC0> disk show smart<br />
Executing: disk show smart<br />
<br />
Smart Method of Enable<br />
Capable Informational Exception Performance Error<br />
B:ID:L Device Exceptions(MRIE) Control Enabled Count<br />
------ ------- ---------------- --------- ----------- ------<br />
0:00:0 Y 6 Y N 0<br />
0:01:0 Y 6 Y N 0<br />
0:02:0 Y 6 Y N 0<br />
0:03:0 Y 6 Y N 0<br />
0:06:0 N<br />
<br />
AAC0> task list<br />
Executing: task list<br />
<br />
Controller Tasks<br />
<br />
TaskId Function Done% Container State Specific1 Specific2<br />
------ -------- ------- --------- ----- --------- ---------<br />
<br />
No tasks currently running on controller<br />
<br />
AAC0> dia sh hi<br />
Executing: diagnostic show history<br />
No switches specified, defaulting to "/current".<br />
<br />
<br />
<br />
*** HISTORY BUFFER FROM CURRENT CONTROLLER RUN ***<br />
<br />
[00]: GetDiskLogEntry: container - 1, entry return 0<br />
[01]: Container 1 started SCRUB task<br />
[02]: Starting Mirror:1 scrub<br />
[03]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[04]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[05]: UpdateDiskLogIndex - Set - container 0, index 1<br />
[06]: GetDiskLogEntry: container - 0, entry return 1<br />
[07]: Container 0 started SCRUB task<br />
[08]: Starting Mirror:0 scrub<br />
[09]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[10]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[11]: Mirror Scrub Container:1 ErrorsFound:0<br />
[12]: Clear disk log: sector - 80, driveno 2<br />
[13]: Clear disk log: sector - 80, driveno 3<br />
[14]: Container 1 completed SCRUB task:<br />
[15]: Mirror Scrub Container:0 ErrorsFound:0<br />
[16]: Clear disk log: sector - 81, driveno 1<br />
[17]: Clear disk log: sector - 81, driveno 0<br />
[18]: Container 0 completed SCRUB task:<br />
[19]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[20]: GetDiskLogEntry: container - 0, entry return 0<br />
[21]: Container 0 started SCRUB task<br />
[22]: Starting Mirror:0 scrub<br />
[23]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[24]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[25]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[26]: GetDiskLogEntry: container - 1, entry return 1<br />
[27]: Container 1 started SCRUB task<br />
[28]: Starting Mirror:1 scrub<br />
[29]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[30]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[31]: Mirror Scrub Container:1 ErrorsFound:0<br />
[32]: Clear disk log: sector - 81, driveno 2<br />
[33]: Clear disk log: sector - 81, driveno 3<br />
[34]: Container 1 completed SCRUB task:<br />
[35]: Mirror Scrub Container:0 ErrorsFound:0<br />
[36]: Clear disk log: sector - 80, driveno 1<br />
[37]: Clear disk log: sector - 80, driveno 0<br />
[38]: Container 0 completed SCRUB task:<br />
[39]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[40]: GetDiskLogEntry: container - 0, entry return 0<br />
[41]: Container 0 started SCRUB task<br />
[42]: Starting Mirror:0 scrub<br />
[43]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[44]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[45]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[46]: GetDiskLogEntry: container - 1, entry return 1<br />
[47]: Container 1 started SCRUB task<br />
[48]: Starting Mirror:1 scrub<br />
[49]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[50]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[51]: Mirror Scrub Container:1 ErrorsFound:0<br />
[52]: Clear disk log: sector - 81, driveno 2<br />
[53]: Clear disk log: sector - 81, driveno 3<br />
[54]: Container 1 completed SCRUB task:<br />
[55]: Mirror Scrub Container:0 ErrorsFound:0<br />
[56]: Clear disk log: sector - 80, driveno 1<br />
[57]: Clear disk log: sector - 80, driveno 0<br />
[58]: Container 0 completed SCRUB task:<br />
[59]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[60]: GetDiskLogEntry: container - 0, entry return 0<br />
[61]: Container 0 started SCRUB task<br />
[62]: Starting Mirror:0 scrub<br />
[63]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[64]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[65]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[66]: GetDiskLogEntry: container - 1, entry return 1<br />
[67]: Container 1 started SCRUB task<br />
[68]: Starting Mirror:1 scrub<br />
[69]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[70]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[71]: Mirror Scrub Container:1 ErrorsFound:0<br />
[72]: Clear disk log: sector - 81, driveno 2<br />
[73]: Clear disk log: sector - 81, driveno 3<br />
[74]: Container 1 completed SCRUB task:<br />
[75]: Mirror Scrub Container:0 ErrorsFound:0<br />
[76]: Clear disk log: sector - 80, driveno 1<br />
[77]: Clear disk log: sector - 80, driveno 0<br />
[78]: Container 0 completed SCRUB task:<br />
[79]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[80]: GetDiskLogEntry: container - 0, entry return 0<br />
[81]: Container 0 started SCRUB task<br />
[82]: Starting Mirror:0 scrub<br />
[83]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[84]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[85]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[86]: GetDiskLogEntry: container - 1, entry return 1<br />
[87]: Container 1 started SCRUB task<br />
[88]: Starting Mirror:1 scrub<br />
[89]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[90]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[91]: Mirror Scrub Container:1 ErrorsFound:0<br />
[92]: Clear disk log: sector - 81, driveno 2<br />
[93]: Clear disk log: sector - 81, driveno 3<br />
[94]: Container 1 completed SCRUB task:<br />
[95]: Mirror Scrub Container:0 ErrorsFound:0<br />
[96]: Clear disk log: sector - 80, driveno 1<br />
[97]: Clear disk log: sector - 80, driveno 0<br />
[98]: Container 0 completed SCRUB task:<br />
[99]:<br />
<br />
========================<br />
History Output Complete.<br />
<br />
AAC0><br />
AAC0> exit<br />
Executing: exit<br />
<br />
press enter when ready to run verify <INS><br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> contai scr 0<br />
Executing: container scrub 0<br />
<br />
AAC0> contai scr 1<br />
Executing: container scrub 1<br />
<br />
AAC0> exit<br />
Executing: exit<br />
<br />
when done run: <br />
<br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
<br />
<br />
Nov 1 10:32:46 mail /kernel: aac0: **Monitor** Container 0 started SCRUB task<br />
Nov 1 10:32:47 mail /kernel: aac0: **Monitor** Container 1 started SCRUB task<br />
<br />
</pre><br />
<br />
Here's an analysis of what we're seeing and what we're looking for:<br />
<br />
<pre><br />
AAC0> container list /f<br />
Executing: container list /full=TRUE<br />
Num Total Oth Chunk Scsi Partition<br />
Creation System<br />
Label Type Size Ctr Size Usage B:ID:L Offset:Size State RO Lk Task Done% Ent<br />
Date Time Files<br />
----- ------ ------ --- ------ ------- ------ ------------- ------- -- -- ------- ------ ---<br />
------ -------- ------<br />
0 Mirror 33.9GB Open 0:01:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:32<br />
/dev/aacd0 mirror0 0:00:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:32<br />
<br />
1 Mirror 33.9GB Open 0:02:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:50<br />
/dev/aacd1 mirror1 0:03:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:50<br />
</pre><br />
<br />
This is showing you the health of the arrays. You're looking for ''Normal'' under the State column, and the absence of a ! in the sector size - sometimes, you'll see this:<br />
64.0KB!33.9GB <br />
<br />
That indicates a problem.<br />
<br />
<pre><br />
AAC0> disk show smart<br />
Executing: disk show smart<br />
<br />
Smart Method of Enable<br />
Capable Informational Exception Performance Error<br />
B:ID:L Device Exceptions(MRIE) Control Enabled Count<br />
------ ------- ---------------- --------- ----------- ------<br />
0:00:0 Y 6 Y N 0<br />
0:01:0 Y 6 Y N 0<br />
0:02:0 Y 6 Y N 0<br />
0:03:0 Y 6 Y N 0<br />
0:06:0 N<br />
</pre><br />
<br />
This shows you a SMART report output. Looking for values in the Error Count column.<br />
<pre><br />
AAC0> task list<br />
Executing: task list<br />
<br />
Controller Tasks<br />
<br />
TaskId Function Done% Container State Specific1 Specific2<br />
------ -------- ------- --------- ----- --------- ---------<br />
<br />
No tasks currently running on controller<br />
</pre><br />
<br />
Look for absence of tasks running- a bad thing would be to see a rebuild or verify running when you didn't initiate it.<br />
<br />
With the history output, you're looking for any anomalies or events since the last time a verify was run. If you see a drive with lots of problems, you may want to take backups before allowing the verify to run since it could replicate errors onto the good drive.<br />
<br />
After you see the history output, it will prompt you to press enter to run the verify. If you're happy with all the output you're seeing- mirror is healthy, history looks good, it's safe to proceed. Otherwise ^C to exit. After hitting enter it will start the verify and start to tail the messages log file (so you can easily see when the verify is complete). Here's what that'll look like:<br />
<br />
<pre><br />
Nov 1 14:38:08 mail /kernel: aac0: **Monitor** Container 1 completed SCRUB task:<br />
Nov 1 14:46:45 mail /kernel: aac0: **Monitor** Container 0 completed SCRUB task:<br />
</pre><br />
<br />
So, putting it all together, after hitting enter to start the verify, you'll see:<br />
<br />
<pre><br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> contai scr 0<br />
Executing: container scrub 0<br />
<br />
AAC0> contai scr 1<br />
Executing: container scrub 1<br />
<br />
AAC0> exit<br />
Executing: exit<br />
<br />
when done run: <br />
<br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
<br />
<br />
Nov 1 10:32:46 mail /kernel: aac0: **Monitor** Container 0 started SCRUB task<br />
Nov 1 10:32:47 mail /kernel: aac0: **Monitor** Container 1 started SCRUB task<br />
<br />
</pre><br />
<br />
When the scrub(s) (verify) are complete - if the server has multiple logical drives, it will run both in parallel - you should exit the tail of the log file (^C) and run:<br />
<br />
<pre><br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
</pre><br />
<br />
Which will show you the diagnostic history, you're looking for the results of the most recent scrub:<br />
<br />
<pre><br />
[100]: Mirror Scrub Container:1 ErrorsFound:0<br />
[101]: Clear disk log: sector - 81, driveno 2<br />
[102]: Clear disk log: sector - 81, driveno 3<br />
[103]: Container 1 completed SCRUB task:<br />
[104]: Mirror Scrub Container:0 ErrorsFound:0<br />
[105]: Clear disk log: sector - 80, driveno 1<br />
[106]: Clear disk log: sector - 80, driveno 0<br />
[107]: Container 0 completed SCRUB task:<br />
</pre><br />
<br />
^C to exit the RAID CLI.<br />
<br />
If you see:<br />
<br />
[104]: Mirror Scrub Container:0 ErrorsFound:5<br />
<br />
You'll want to rerun the verify on that drive till it shows 0, or perhaps replace the drive- you should be able to see from the output which drive had the problem.<br />
<br />
Depending on the size and how busy the drive is, the verify can take anywhere from an hour to the better part of a day.<br />
<br />
You will notice that the diagnostic history is not shown on our modern adaptec cards (i.e. any adaptec card not in a Dell 2450). The reason for this is the history is never cleared, so there's simply too much data to show and it just crashes the CLI. So, don't bother trying to see it...which does make it hard to see if there are problems going on, so you just need to watch the scrub to see it goes to 100%. You will also notice that on some servers there's no tail of messages. Again, this is cause no data is shown there about the completion of the scrub. The thing to do here is to go into the CLI and continue to <tt>show tasks</tt> to monitor scrub progress.<br />
<br />
See [[RAIC_CLI#Adaptec|Adaptec RAID CLI Reference]] for more details on how to use the CLI<br />
<br />
=== DELL (LSI-based) SAS controllers ===<br />
<br />
Here's what the output looks like when running verify.sh on a LSI-based card:<br />
<br />
<pre><br />
jail2 /mnt/data2# sh /root/verify.sh<br />
<br />
Adapter #0<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 0<br />
Device Id: 0<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018396142<br />
SAS Address(1): 0x0<br />
Connected Port Number: 0(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DESN<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 1<br />
Device Id: 1<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018395db2<br />
SAS Address(1): 0x0<br />
Connected Port Number: 1(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DERV<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 2<br />
Device Id: 2<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50006eece89<br />
SAS Address(1): 0x0<br />
Connected Port Number: 2(path0)<br />
Inquiry Data: SEAGATE ST3300555SS T2113LM4BFBZ<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 3<br />
Device Id: 3<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50006eee035<br />
SAS Address(1): 0x0<br />
Connected Port Number: 3(path0)<br />
Inquiry Data: SEAGATE ST3300555SS T2113LM4BGF7<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 4<br />
Device Id: 4<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50004bd7ea5<br />
SAS Address(1): 0x0<br />
Connected Port Number: 4(path0)<br />
Inquiry Data: SEAGATE ST3300656SS HS093QP0G8SW<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 5<br />
Device Id: 5<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e01f1c4112<br />
SAS Address(1): 0x0<br />
Connected Port Number: 5(path0)<br />
Inquiry Data: FUJITSU MBA3300RC D306BJ15P9201W06<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 0 (Target Id: 0)<br />
Name:<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:139392MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 1 (Target Id: 1)<br />
Name:MIRROR1<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 2 (Target Id: 2)<br />
Name:MIRROR2<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
Battery FRU : N/A<br />
Battery Warning : Enabled<br />
Memory Correctable Errors : 0<br />
Memory Uncorrectable Errors : 0<br />
BBU : Present<br />
BBU : Yes<br />
Cache When BBU Bad : Disabled<br />
press enter when ready to run verify<br />
<br />
</pre><br />
<br />
Before pressing enter, here's what we're looking for:<br />
<br />
<pre><br />
Enclosure Device ID: 32<br />
Slot Number: 0<br />
Device Id: 0<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018396142<br />
SAS Address(1): 0x0<br />
Connected Port Number: 0(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DESN<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
</pre><br />
<br />
This is the output shown for each physical drive in the system. We're looking to confirm it's Firmware state is ''Online'', and Media Error Count, Other Error Count, and Predictive Failure Count are all zero (or near zero).<br />
<br />
<pre><br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 1 (Target Id: 1)<br />
Name:MIRROR1<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
</pre><br />
<br />
This is the output for each logical drive. We're looking for State ''Optimal''. Also confirm <tt>Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU</tt><br />
<br />
<pre><br />
Exit Code: 0x00<br />
Battery FRU : N/A<br />
Battery Warning : Enabled<br />
Memory Correctable Errors : 0<br />
Memory Uncorrectable Errors : 0<br />
BBU : Present<br />
BBU : Yes<br />
Cache When BBU Bad : Disabled<br />
</pre><br />
<br />
Confirm that the battery is present and error-free.<br />
<br />
If all that checks out, you're ready to proceed with the verify. After pressing enter, the verify is started and here's what you see:<br />
<pre><br />
Start Check Consistency on Virtual Drive 0 (target id: 0) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Start Check Consistency on Virtual Drive 1 (target id: 1) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Start Check Consistency on Virtual Drive 2 (target id: 2) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Check Consistency<br />
<br />
Progress of Virtual Drives...<br />
<br />
Virtual Drive # Percent Complete Time Elps<br />
0 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:03<br />
1 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:02<br />
2 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:01<br />
<br />
Press <ESC> key to quit...<br />
</pre><br />
<br />
The progress for each drive is displayed until all drives have completed the verify. We just want to make sure that each drive goes to completion. No followup is needed...though there probably is a log or history where we can get more info.<br />
<br />
You will notice that jail7 does not run a verify- that's on purpose. The last time we tried this it crashed the system. So, this must be run from the BIOS (take the system offline for a couple hours).<br />
<br />
See [[RAIC_CLI#LSI|LSI RAID CLI Reference]] for more details on how to use the CLI<br />
<br />
=== LSI-based controllers (megaraid) ===<br />
<br />
There is a CLI for this however it's easier to do this with a curses GUI app: megaraid<br />
<br />
Currently only on these servers: virt15, virt16, and firewall2<br />
<br />
To run:<br />
<br />
# cd /usr/local/sbin/; megamgr<br />
<br />
Main menu:<br />
<pre>²ÚÄÄManagement MenuÄÄ¿²<br />
²³ Configure ³²<br />
²³ Initialize ³²<br />
²³ Objects ³²<br />
²³ Rebuild ³²<br />
²³ Check Consistency ³²<br />
²³ Advanced Menu ³²<br />
²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²</pre><br />
<br />
Before you check consistency, make sure the arrays are healthy.<br />
<br />
Objects -> Physical Drive<br />
<br />
Then look to make sure they're all <tt>ONLIN</tt><br />
<pre><br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄObjects - PHYSICAL DRIVE SELECTION MENUÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²<br />
²²²²²²²²³ ³²²²²²²<br />
²²²²²²²²³ Channel-1 ³²²²²²²<br />
²ÚÄÄMana³ ID ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»x ³²²²²²²<br />
²³ Confi³ 0º* ONLIN A01-01º ³²²²²²²<br />
²³ Initi³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²³ Objec³ 1º* ONLIN A01-02º ³²²²²²²<br />
²³ Rebui³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²³ Check³ 2º* ONLIN A02-01º ³²²²²²²<br />
²³ Advan³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²ÀÄÄÄÄÄij 3º* ONLIN A02-02º ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 4º* ONLIN A03-01º ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 5º* ONLIN A03-02ºþ ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 6º* º ³²²²²²²<br />
²²²²²²²²³ ÈÍÍÍÍÍÍÍÍÍÍÍÍÍͼx ³²²²²²²<br />
²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²<br />
²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²²²<br />
²²²²²²²²³Ch-1 ID-5 DISK 140013MB SEAGATE ST3146707LC 0003 ³²²²²²²²²<br />
²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²<br />
</pre><br />
<br />
<br />
Once that's done, hit escape once then the back arror to move back to the Objects menu.<br />
So you select Objects -> Logical Drive -> Logical Drive 1 -> Check Consistency -> YES<br />
<br />
<pre><br />
<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÚÄLogical Drives(02)Ä¿²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ Logical Drive 1 ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²ÚÄÄÄÄObjectsÄÄij Logical Drive 2 ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²ÚÄÄManagemen³ Adapter ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Configure ³ Logical Drive ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Initialize³ Physical Drive ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Objects ³ Channel ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Rebuild ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Check Consistency ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Advanced Menu ³²²²²²²²ÚÄÄÄÄLogical Drive 1ÄÄÄÄÄ¿²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²³ Initialize ÚÄCheck Consistency-1 ?Ä¿²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ Check Consiste³ YES ³²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ View/Update Pa³ NO ³²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³Select YES Or NO³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
</pre><br />
<br />
Then watch the progress. When done, escape back to Logical Drive then repeat for Logical Drive 2. If you ^C or accidentally escape out, you can come back in running the same commands and watch the progress again (it won't restart).<br />
<br />
You can exit megamgr by escaping out or ^C<br />
<br />
=== 3ware ===<br />
<br />
We are using 3ware controllers on backup1 & backup2. Running the verify script will give you different output based on the type of controller:<br />
<br />
<pre><br />
backup2 /d2# sh /root/verify.sh<br />
Controller: c0<br />
-------------<br />
Driver: 1.50.01.002<br />
Model: 7500-8<br />
FW: FE7X 1.05.00.068<br />
BIOS: BE7X 1.08.00.048<br />
Monitor: ME7X 1.01.00.040<br />
Serial #: F11605A3180172<br />
PCB: Rev3<br />
PCHIP: 1.30-33<br />
ACHIP: 3.20<br />
<br />
<br />
# of units: 3<br />
Unit 0: JBOD 186.31 GB ( 390721968 blocks): OK<br />
Unit 1: RAID 5 465.77 GB ( 976790016 blocks): DEGRADED<br />
Unit 5: RAID 5 698.65 GB ( 1465185024 blocks): DEGRADED<br />
<br />
# of ports: 8<br />
Port 0: WDC WD2000JB-00KFA0 WD-WCAMT1451690 186.31 GB (390721968 blocks): OK(unit 0)<br />
Port 1: WDC WD2500JB-00GVC0 WD-WCAL78219488 232.88 GB (488397168 blocks): OK(unit 1)<br />
Port 2: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT)<br />
Port 3: WDC WD2500JB-00GVC0 WD-WMAL73882417 232.88 GB (488397168 blocks): OK(unit 1)<br />
Port 4: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT)<br />
Port 5: WDC WD2500JB-00GVA0 WD-WMAL71338097 232.88 GB (488397168 blocks): OK(unit 5)<br />
Port 6: WDC WD2500JB-32EVA0 WD-WMAEH1301595 232.88 GB (488397168 blocks): OK(unit 5)<br />
Port 7: WDC WD2500JB-00GVC0 WD-WCAL78165566 232.88 GB (488397168 blocks): OK(unit 5)<br />
Controller: c1<br />
-------------<br />
Driver: 1.50.01.002<br />
Model: 7500-8<br />
FW: FE7X 1.05.00.068<br />
BIOS: BE7X 1.08.00.048<br />
Monitor: ME7X 1.01.00.040<br />
Serial #: F11605A3180167<br />
PCB: Rev3<br />
PCHIP: 1.30-33<br />
ACHIP: 3.20<br />
<br />
<br />
# of units: 2<br />
Unit 0: RAID 5 698.65 GB ( 1465185024 blocks): OK<br />
Unit 4: RAID 5 698.65 GB ( 1465185024 blocks): OK<br />
<br />
# of ports: 8<br />
Port 0: WDC WD2500JB-00GVA0 WD-WMAL71301258 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 1: WDC WD2500JB-00GVA0 WD-WMAL71322705 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 2: WDC WD2500JB-00GVA0 WD-WMAL71945050 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 3: WDC WD2500JB-00GVA0 WD-WMAL71316201 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 4: WDC WD2500JB-00GVC0 WD-WCAL78323749 232.88 GB (488397168 blocks): OK(unit 4)<br />
Port 5: WDC WD3200AAJB-00J3A0 WD-WCAV2V689068 298.09 GB (625142448 blocks): OK(unit 4)<br />
Port 6: WDC WD2500JB-00GVC0 WD-WCAL78234420 232.88 GB (488397168 blocks): OK(unit 4)<br />
Port 7: WDC WD2500JB-00GVC0 WD-WCAL78592213 232.88 GB (488397168 blocks): OK(unit 4)<br />
backup2 /d2#<br />
</pre><br />
<br />
On backup2 look for all ok, no verify.<br />
<br />
<pre><br />
[root@backup3 ~]# sh /root/verify.sh<br />
/c2 Driver Version = 1.26.02.002<br />
/c2 Model = 8006-2LP<br />
/c2 Available Memory = 512KB<br />
/c2 Firmware Version = FE8S 1.05.00.068<br />
/c2 Bios Version = BE7X 1.08.00.048<br />
/c2 Boot Loader Version = ME7X 1.01.00.040<br />
/c2 Serial Number = L018501C6481395<br />
/c2 PCB Version = Rev5<br />
/c2 PCHIP Version = 1.30-66<br />
/c2 ACHIP Version = 3.20<br />
/c2 Total Optimal Units = 1<br />
/c2 Not Optimal Units = 0<br />
<br />
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u1 RAID-1 OK - - - 931.512 ON -<br />
<br />
Port Status Unit Size Blocks Serial<br />
---------------------------------------------------------------<br />
p0 OK u1 931.51 GB 1953525168 WD-WMAW31148820<br />
p1 OK u1 931.51 GB 1953525168 WD-WCATR0277515<br />
<br />
<br />
Ctl Date Severity Alarm Message<br />
------------------------------------------------------------------------------<br />
<br />
Sending start verify message to /c2/u1 ... Done.<br />
<br />
<br />
when done run:<br />
tw_cli /c2 show alarms<br />
<br />
[root@backup3 ~]#<br />
</pre><br />
<br />
Automatically starts the verify, just run <tt>tw_cli /c2 show alarms</tt> as instructed to see the results of the verify.<br />
<br />
<pre><br />
[root@backup1 /data/deprecated]# sh /root/verify.sh<br />
/c0 Driver Version = 2.26.02.010<br />
/c0 Model = 9650SE-8LPML<br />
/c0 Available Memory = 224MB<br />
/c0 Firmware Version = FE9X 4.06.00.004<br />
/c0 Bios Version = BE9X 4.05.00.015<br />
/c0 Boot Loader Version = BL9X 3.08.00.001<br />
/c0 Serial Number = L326025A8270177<br />
/c0 PCB Version = Rev 032<br />
/c0 PCHIP Version = 2.00<br />
/c0 ACHIP Version = 1.90<br />
/c0 Number of Ports = 8<br />
/c0 Number of Drives = 6<br />
/c0 Number of Units = 1<br />
/c0 Total Optimal Units = 1<br />
/c0 Not Optimal Units = 0<br />
/c0 JBOD Export Policy = off<br />
/c0 Disk Spinup Policy = 1<br />
/c0 Spinup Stagger Time Policy (sec) = 1<br />
/c0 Auto-Carving Policy = off<br />
/c0 Auto-Carving Size = 2048 GB<br />
/c0 Auto-Rebuild Policy = on<br />
/c0 Controller Bus Type = PCIe<br />
/c0 Controller Bus Width = 1 lane<br />
/c0 Controller Bus Speed = 2.5 Gbps/lane<br />
<br />
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u0 RAID-5 OK - - 64K 4656.56 ON ON<br />
<br />
Port Status Unit Size Blocks Serial<br />
---------------------------------------------------------------<br />
p0 OK u0 931.51 GB 1953525168 9QJ1Y017<br />
p1 DEVICE-ERROR u0 931.51 GB 1953525168 9QJ1ZN07<br />
p2 OK u0 931.51 GB 1953525168 9QJ2XK1R<br />
p3 OK u0 931.51 GB 1953525168 9QJ2010B<br />
p4 OK u0 1.36 TB 2930277168 6XW0L36T<br />
p5 OK u0 931.51 GB 1953525168 WD-WMATV2444836<br />
p6 NOT-PRESENT - - - -<br />
p7 NOT-PRESENT - - - -<br />
<br />
<br />
Ctl Date Severity Alarm Message<br />
------------------------------------------------------------------------------<br />
c0 [Sat May 12 11:27:15 2012] WARNING Sector repair completed: port=0, LBA=0x6AE571C<br />
c0 [Sat May 12 19:16:21 2012] WARNING Sector repair completed: port=1, LBA=0x40E62A23<br />
c0 [Sat May 12 21:40:56 2012] INFO Verify completed: unit=0<br />
c0 [Mon May 14 00:53:53 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFA7<br />
c0 [Mon May 14 00:58:21 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAA<br />
c0 [Mon May 14 04:35:13 2012] WARNING Sector repair completed: port=0, LBA=0x8FEF2CF<br />
c0 [Mon May 14 04:38:22 2012] WARNING Sector repair completed: port=0, LBA=0x8FEF2D1<br />
c0 [Tue May 15 22:53:46 2012] WARNING Sector repair completed: port=0, LBA=0x13C2622<br />
c0 [Wed May 16 00:39:31 2012] WARNING Sector repair completed: port=0, LBA=0x365A67F<br />
c0 [Wed May 16 00:39:37 2012] WARNING Sector repair completed: port=0, LBA=0x365A685<br />
c0 [Wed May 16 00:47:18 2012] WARNING Sector repair completed: port=0, LBA=0x365A687<br />
c0 [Sat May 19 00:01:44 2012] INFO Verify started: unit=0<br />
c0 [Sat May 19 04:46:20 2012] WARNING Sector repair completed: port=0, LBA=0x365A68E<br />
c0 [Sat May 19 13:37:06 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAC<br />
c0 [Sat May 19 13:37:28 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAE<br />
c0 [Sat May 19 13:37:47 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFB1<br />
c0 [Sat May 19 13:38:00 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFB3<br />
c0 [Sat May 19 21:47:45 2012] INFO Verify completed: unit=0<br />
c0 [Wed May 23 12:21:41 2012] INFO Cache synchronization completed: unit=0<br />
c0 [Fri May 25 00:08:19 2012] WARNING Sector repair completed: port=0, LBA=0x12DA76C<br />
c0 [Fri May 25 00:08:34 2012] WARNING Sector repair completed: port=0, LBA=0x12E4901<br />
c0 [Fri May 25 00:09:33 2012] WARNING Sector repair completed: port=0, LBA=0x12DA773<br />
c0 [Fri May 25 00:39:12 2012] WARNING Sector repair completed: port=0, LBA=0x42C597B<br />
c0 [Sat May 26 00:01:45 2012] INFO Verify started: unit=0<br />
c0 [Sat May 26 00:42:05 2012] WARNING Sector repair completed: port=1, LBA=0x323C1AC<br />
c0 [Sat May 26 00:51:43 2012] WARNING Sector repair completed: port=1, LBA=0x323C1AE<br />
c0 [Sat May 26 01:54:21 2012] WARNING Sector repair completed: port=1, LBA=0x2F0D302<br />
c0 [Sat May 26 02:06:38 2012] WARNING Sector repair completed: port=0, LBA=0x12DA777<br />
c0 [Sat May 26 02:07:21 2012] WARNING Sector repair completed: port=0, LBA=0x12E48FE<br />
c0 [Sat May 26 04:20:00 2012] WARNING Sector repair completed: port=1, LBA=0x2F0D306<br />
c0 [Sat May 26 04:32:58 2012] WARNING Sector repair completed: port=1, LBA=0x323C1B1<br />
c0 [Sat May 26 04:33:21 2012] WARNING Sector repair completed: port=1, LBA=0x323C1B3<br />
c0 [Sat May 26 04:33:44 2012] WARNING Sector repair completed: port=1, LBA=0x323C1BA<br />
c0 [Sat May 26 05:24:07 2012] WARNING Sector repair completed: port=1, LBA=0x3F83862<br />
c0 [Sat May 26 05:25:09 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 06:08:13 2012] WARNING Sector repair completed: port=0, LBA=0x4CDC6A2<br />
c0 [Sat May 26 09:49:35 2012] WARNING Sector repair completed: port=1, LBA=0x6CACD4A<br />
c0 [Sat May 26 18:10:44 2012] WARNING Sector repair completed: port=1, LBA=0x18F425EA<br />
c0 [Sat May 26 19:45:40 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:22:52 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:15 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:22 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:35 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:41 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:49 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:57 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:24:02 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:54:41 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 22:00:30 2012] INFO Verify completed: unit=0<br />
c0 [Sat Jun 2 00:01:43 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 2 00:30:17 2012] WARNING Sector repair completed: port=0, LBA=0x2B911E4<br />
c0 [Sat Jun 2 00:50:57 2012] WARNING Sector repair completed: port=0, LBA=0x5A807CA6<br />
c0 [Sat Jun 2 04:13:13 2012] WARNING Sector repair completed: port=0, LBA=0x2D18291<br />
c0 [Sat Jun 2 04:13:35 2012] WARNING Sector repair completed: port=0, LBA=0x2D1829F<br />
c0 [Sat Jun 2 21:48:02 2012] INFO Verify completed: unit=0<br />
c0 [Mon Jun 4 04:40:34 2012] WARNING Sector repair completed: port=1, LBA=0x4AF8098F<br />
c0 [Tue Jun 5 00:28:19 2012] WARNING Sector repair completed: port=1, LBA=0x263C5CD<br />
c0 [Tue Jun 5 00:33:06 2012] WARNING Sector repair completed: port=1, LBA=0x263C5CF<br />
c0 [Thu Jun 7 00:34:27 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B5F<br />
c0 [Thu Jun 7 00:38:50 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B61<br />
c0 [Fri Jun 8 00:07:13 2012] WARNING Sector repair completed: port=0, LBA=0xC131F6B<br />
c0 [Sat Jun 9 00:01:41 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 9 00:29:11 2012] WARNING Sector repair completed: port=0, LBA=0x6C7614D<br />
c0 [Sat Jun 9 00:38:25 2012] WARNING Sector repair completed: port=0, LBA=0x6C76152<br />
c0 [Sat Jun 9 04:02:30 2012] WARNING Sector repair completed: port=1, LBA=0x263C5D1<br />
c0 [Sat Jun 9 04:02:52 2012] WARNING Sector repair completed: port=1, LBA=0x263C5D3<br />
c0 [Sat Jun 9 04:07:32 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E12<br />
c0 [Sat Jun 9 04:07:57 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E15<br />
c0 [Sat Jun 9 04:08:16 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E17<br />
c0 [Sat Jun 9 04:08:45 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E19<br />
c0 [Sat Jun 9 04:15:04 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B64<br />
c0 [Sat Jun 9 04:15:26 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B66<br />
c0 [Sat Jun 9 04:15:45 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B68<br />
c0 [Sat Jun 9 04:15:59 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B6C<br />
c0 [Sat Jun 9 04:16:13 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B6E<br />
c0 [Sat Jun 9 21:48:52 2012] INFO Verify completed: unit=0<br />
c0 [Thu Jun 14 00:40:10 2012] WARNING Sector repair completed: port=0, LBA=0x334F14B<br />
c0 [Sat Jun 16 00:01:38 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 16 21:16:19 2012] INFO Verify completed: unit=0<br />
c0 [Tue Jun 19 02:03:43 2012] WARNING Sector repair completed: port=1, LBA=0xFE41EAD<br />
c0 [Wed Jun 20 02:30:02 2012] WARNING Sector repair completed: port=1, LBA=0xD99145C<br />
c0 [Sat Jun 23 00:01:36 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 23 04:27:04 2012] WARNING Sector repair completed: port=1, LBA=0x2FAD311<br />
c0 [Sat Jun 23 06:52:38 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC8D<br />
c0 [Sat Jun 23 06:53:03 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC91<br />
c0 [Sat Jun 23 06:53:21 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC94<br />
c0 [Sat Jun 23 17:00:22 2012] WARNING Sector repair completed: port=1, LBA=0xF9AC7C9<br />
c0 [Sat Jun 23 21:15:19 2012] INFO Verify completed: unit=0<br />
c0 [Sat Jun 30 00:01:34 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 30 05:24:13 2012] WARNING Sector repair completed: port=0, LBA=0x3FAA9E7<br />
c0 [Sat Jun 30 14:49:39 2012] WARNING Sector repair completed: port=1, LBA=0x869931C<br />
c0 [Sat Jun 30 21:31:05 2012] INFO Verify completed: unit=0<br />
c0 [Tue Jul 3 03:40:25 2012] WARNING Sector repair completed: port=1, LBA=0xD36C7F7<br />
c0 [Fri Jul 6 02:50:18 2012] WARNING Sector repair completed: port=1, LBA=0x3562470<br />
c0 [Fri Jul 6 22:18:26 2012] WARNING Sector repair completed: port=1, LBA=0x3563173<br />
c0 [Sat Jul 7 00:01:31 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 7 00:50:16 2012] WARNING Sector repair completed: port=0, LBA=0x76EE88<br />
c0 [Sat Jul 7 00:50:39 2012] WARNING Sector repair completed: port=0, LBA=0x76EE8F<br />
c0 [Sat Jul 7 21:39:36 2012] INFO Verify completed: unit=0<br />
c0 [Sun Jul 8 02:51:05 2012] WARNING Sector repair completed: port=0, LBA=0x67759D<br />
c0 [Sun Jul 8 02:53:55 2012] WARNING Sector repair completed: port=0, LBA=0x67759B<br />
c0 [Tue Jul 10 16:17:21 2012] WARNING Sector repair completed: port=0, LBA=0x15C8C695<br />
c0 [Wed Jul 11 22:51:22 2012] WARNING Sector repair completed: port=1, LBA=0x355BBD0<br />
c0 [Sat Jul 14 00:01:28 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 14 01:33:40 2012] WARNING Sector repair completed: port=1, LBA=0x1333BCF4<br />
c0 [Sat Jul 14 03:36:23 2012] WARNING Sector repair completed: port=1, LBA=0x2174773<br />
c0 [Sat Jul 14 11:26:44 2012] WARNING Sector repair completed: port=0, LBA=0x7429AB7<br />
c0 [Sat Jul 14 16:53:50 2012] WARNING Sector repair completed: port=1, LBA=0xA17EB3F<br />
c0 [Sat Jul 14 21:19:25 2012] INFO Verify completed: unit=0<br />
c0 [Wed Jul 18 05:08:47 2012] WARNING Sector repair completed: port=1, LBA=0x17D62EDC<br />
c0 [Wed Jul 18 05:14:15 2012] WARNING Sector repair completed: port=1, LBA=0x17D62EE1<br />
c0 [Thu Jul 19 03:24:59 2012] WARNING Sector repair completed: port=0, LBA=0x7733C3D<br />
c0 [Thu Jul 19 03:25:20 2012] WARNING Sector repair completed: port=0, LBA=0x773CEA5<br />
c0 [Thu Jul 19 03:28:16 2012] WARNING Sector repair completed: port=0, LBA=0x7733C42<br />
c0 [Thu Jul 19 03:28:41 2012] WARNING Sector repair completed: port=0, LBA=0x773CEAF<br />
c0 [Sat Jul 21 00:01:26 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 21 03:07:30 2012] WARNING Sector repair completed: port=1, LBA=0x1CC6936<br />
c0 [Sat Jul 21 03:07:52 2012] WARNING Sector repair completed: port=1, LBA=0x1CC6938<br />
c0 [Sat Jul 21 03:08:11 2012] WARNING Sector repair completed: port=1, LBA=0x1CC693A<br />
c0 [Sat Jul 21 16:43:56 2012] WARNING Sector repair completed: port=0, LBA=0xD04C914<br />
c0 [Sat Jul 21 16:45:31 2012] WARNING Sector repair completed: port=1, LBA=0xD456973<br />
c0 [Sat Jul 21 21:14:29 2012] INFO Verify completed: unit=0<br />
c0 [Wed Jul 25 03:37:25 2012] WARNING Sector repair completed: port=0, LBA=0x1F8E6C43<br />
c0 [Sat Jul 28 00:01:24 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 28 01:45:27 2012] WARNING Sector repair completed: port=0, LBA=0x11584AD<br />
c0 [Sat Jul 28 18:54:25 2012] WARNING Sector repair completed: port=1, LBA=0x447C3E6C<br />
c0 [Sat Jul 28 21:13:46 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 1 03:20:11 2012] WARNING Sector repair completed: port=0, LBA=0x805FEF<br />
c0 [Fri Aug 3 00:50:03 2012] WARNING Sector repair completed: port=0, LBA=0xCED0ACA<br />
c0 [Sat Aug 4 00:01:22 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 4 00:52:51 2012] WARNING Sector repair completed: port=0, LBA=0x805FF3<br />
c0 [Sat Aug 4 00:53:14 2012] WARNING Sector repair completed: port=0, LBA=0x805FF5<br />
c0 [Sat Aug 4 00:53:33 2012] WARNING Sector repair completed: port=0, LBA=0x805FF7<br />
c0 [Sat Aug 4 00:53:47 2012] WARNING Sector repair completed: port=0, LBA=0x805FF9<br />
c0 [Sat Aug 4 00:54:00 2012] WARNING Sector repair completed: port=0, LBA=0x805FFB<br />
c0 [Sat Aug 4 00:54:14 2012] WARNING Sector repair completed: port=0, LBA=0x805FFD<br />
c0 [Sat Aug 4 00:54:27 2012] WARNING Sector repair completed: port=0, LBA=0x805FFF<br />
c0 [Sat Aug 4 04:43:12 2012] WARNING Sector repair completed: port=1, LBA=0x16974289<br />
c0 [Sat Aug 4 04:58:17 2012] WARNING Sector repair completed: port=1, LBA=0x1697428E<br />
c0 [Sat Aug 4 20:54:53 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 8 03:21:55 2012] ERROR Drive timeout detected: port=1<br />
c0 [Wed Aug 8 15:31:44 2012] WARNING Sector repair completed: port=0, LBA=0x1A366CD3<br />
c0 [Sat Aug 11 00:01:21 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 11 20:40:51 2012] INFO Verify completed: unit=0<br />
c0 [Thu Aug 16 05:10:55 2012] WARNING Sector repair completed: port=0, LBA=0x1C22593<br />
c0 [Sat Aug 18 00:01:18 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 18 03:00:20 2012] WARNING Sector repair completed: port=0, LBA=0x1C225A5<br />
c0 [Sat Aug 18 03:43:00 2012] WARNING Sector repair completed: port=1, LBA=0x23EE91E<br />
c0 [Sat Aug 18 03:43:23 2012] WARNING Sector repair completed: port=1, LBA=0x23EE920<br />
c0 [Sat Aug 18 17:00:06 2012] WARNING Sector repair completed: port=1, LBA=0x137D066A<br />
c0 [Sat Aug 18 17:00:29 2012] WARNING Sector repair completed: port=1, LBA=0x137D066D<br />
c0 [Sat Aug 18 21:13:01 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 22 01:36:08 2012] WARNING Sector repair completed: port=0, LBA=0x2560A0F<br />
c0 [Wed Aug 22 01:37:42 2012] WARNING Sector repair completed: port=0, LBA=0x2560A13<br />
c0 [Fri Aug 24 04:01:36 2012] WARNING Sector repair completed: port=1, LBA=0x55C1A5DF<br />
c0 [Fri Aug 24 05:02:06 2012] WARNING Sector repair completed: port=1, LBA=0xCE3378A<br />
c0 [Sat Aug 25 00:01:17 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 25 00:31:06 2012] WARNING Sector repair completed: port=1, LBA=0x50F65D<br />
c0 [Sat Aug 25 00:39:52 2012] WARNING Sector repair completed: port=0, LBA=0x678FF4<br />
c0 [Sat Aug 25 03:43:15 2012] WARNING Sector repair completed: port=0, LBA=0x2560A15<br />
c0 [Sat Aug 25 03:43:39 2012] WARNING Sector repair completed: port=0, LBA=0x2560A19<br />
c0 [Sat Aug 25 03:43:58 2012] WARNING Sector repair completed: port=0, LBA=0x2560A1B<br />
c0 [Sat Aug 25 03:44:30 2012] WARNING Sector repair completed: port=0, LBA=0x2560A21<br />
c0 [Sat Aug 25 20:58:14 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 29 04:57:15 2012] WARNING Sector repair completed: port=1, LBA=0xF3957EB<br />
c0 [Sat Sep 1 00:01:15 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 1 03:21:52 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC86<br />
c0 [Sat Sep 1 03:22:15 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC88<br />
c0 [Sat Sep 1 03:22:34 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC8A<br />
c0 [Sat Sep 1 03:22:47 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC8C<br />
c0 [Sat Sep 1 17:17:22 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD1<br />
c0 [Sat Sep 1 17:17:45 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD3<br />
c0 [Sat Sep 1 17:18:04 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD5<br />
c0 [Sat Sep 1 21:36:56 2012] INFO Verify completed: unit=0<br />
c0 [Thu Sep 6 00:07:30 2012] WARNING Sector repair completed: port=0, LBA=0xDA3C64B<br />
c0 [Thu Sep 6 00:32:56 2012] WARNING Sector repair completed: port=1, LBA=0x6BBA816<br />
c0 [Sat Sep 8 00:01:13 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 8 00:09:56 2012] WARNING Sector repair completed: port=0, LBA=0xDEBC958<br />
c0 [Sat Sep 8 04:38:45 2012] WARNING Sector repair completed: port=0, LBA=0x38D254F<br />
c0 [Sat Sep 8 20:44:50 2012] INFO Verify completed: unit=0<br />
c0 [Mon Sep 10 01:26:34 2012] WARNING Sector repair completed: port=1, LBA=0xFFD8D5E<br />
c0 [Wed Sep 12 00:33:48 2012] WARNING Sector repair completed: port=1, LBA=0xE8DB928<br />
c0 [Wed Sep 12 00:36:33 2012] WARNING Sector repair completed: port=1, LBA=0x6D49411<br />
c0 [Fri Sep 14 01:59:39 2012] WARNING Sector repair completed: port=0, LBA=0x1467F1C<br />
c0 [Fri Sep 14 02:08:27 2012] WARNING Sector repair completed: port=1, LBA=0x14C8ABD<br />
c0 [Fri Sep 14 03:54:47 2012] WARNING Sector repair completed: port=0, LBA=0x1580C915<br />
c0 [Sat Sep 15 00:01:11 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 15 02:38:14 2012] WARNING Sector repair completed: port=0, LBA=0x1C178973<br />
c0 [Sat Sep 15 02:59:02 2012] WARNING Sector repair completed: port=0, LBA=0x1C178975<br />
c0 [Sat Sep 15 04:47:08 2012] WARNING Sector repair completed: port=0, LBA=0x3FA0356<br />
c0 [Sat Sep 15 04:47:31 2012] WARNING Sector repair completed: port=0, LBA=0x3FA0359<br />
c0 [Sat Sep 15 10:41:59 2012] WARNING Sector repair completed: port=0, LBA=0x6DFD1EC<br />
c0 [Sat Sep 15 13:25:23 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD100<br />
c0 [Sat Sep 15 13:25:31 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD104<br />
c0 [Sat Sep 15 13:25:54 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD106<br />
c0 [Sat Sep 15 17:10:50 2012] WARNING Sector repair completed: port=0, LBA=0x1C178977<br />
c0 [Sat Sep 15 20:59:57 2012] INFO Verify completed: unit=0<br />
c0 [Tue Sep 18 01:17:18 2012] WARNING Sector repair completed: port=1, LBA=0x803B05B<br />
c0 [Sat Sep 22 00:01:10 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 22 20:54:31 2012] INFO Verify completed: unit=0<br />
c0 [Tue Sep 25 01:56:47 2012] WARNING Sector repair completed: port=0, LBA=0x26E3909<br />
c0 [Sat Sep 29 00:01:08 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 29 02:04:14 2012] WARNING Sector repair completed: port=0, LBA=0x146AC03<br />
c0 [Sat Sep 29 10:58:39 2012] WARNING Sector repair completed: port=0, LBA=0x6D4EB0E<br />
c0 [Sat Sep 29 10:59:02 2012] WARNING Sector repair completed: port=0, LBA=0x6D4EB14<br />
c0 [Sat Sep 29 11:22:44 2012] WARNING Sector repair completed: port=0, LBA=0x6F79623<br />
c0 [Sat Sep 29 13:50:48 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D65E<br />
c0 [Sat Sep 29 13:51:11 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D661<br />
c0 [Sat Sep 29 13:51:30 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D663<br />
c0 [Sat Sep 29 20:57:34 2012] INFO Verify completed: unit=0<br />
c0 [Mon Oct 1 04:47:24 2012] WARNING Sector repair completed: port=0, LBA=0xC5BC6F2<br />
c0 [Tue Oct 2 02:00:27 2012] WARNING Sector repair completed: port=0, LBA=0x1547667<br />
c0 [Tue Oct 2 02:01:56 2012] WARNING Sector repair completed: port=0, LBA=0x154766F<br />
c0 [Tue Oct 2 05:02:31 2012] WARNING Sector repair completed: port=1, LBA=0xD67D054<br />
c0 [Tue Oct 2 05:04:14 2012] WARNING Sector repair completed: port=1, LBA=0xD67D056<br />
c0 [Wed Oct 3 01:22:12 2012] WARNING Sector repair completed: port=1, LBA=0x12AAF8CA<br />
c0 [Thu Oct 4 04:29:22 2012] WARNING Sector repair completed: port=0, LBA=0x13E6F992<br />
c0 [Thu Oct 4 05:10:51 2012] WARNING Sector repair completed: port=0, LBA=0x1C252A4<br />
c0 [Sat Oct 6 00:01:07 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 6 19:41:18 2012] WARNING Sector repair completed: port=1, LBA=0x5A5C3AE8<br />
c0 [Sat Oct 6 21:01:05 2012] INFO Verify completed: unit=0<br />
c0 [Mon Oct 8 00:32:06 2012] WARNING Sector repair completed: port=0, LBA=0x6C60D3E<br />
c0 [Tue Oct 9 03:51:03 2012] WARNING Sector repair completed: port=1, LBA=0x89B5EC9<br />
c0 [Thu Oct 11 04:21:17 2012] WARNING Sector repair completed: port=1, LBA=0x13F85833<br />
c0 [Sat Oct 13 00:01:05 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 13 05:12:40 2012] WARNING Sector repair completed: port=0, LBA=0x3FA5134<br />
c0 [Sat Oct 13 21:08:35 2012] INFO Verify completed: unit=0<br />
c0 [Tue Oct 16 03:53:50 2012] WARNING Sector repair completed: port=1, LBA=0x148AA1BD<br />
c0 [Thu Oct 18 03:20:30 2012] WARNING Sector repair completed: port=0, LBA=0x1C8DABCB<br />
c0 [Thu Oct 18 04:52:50 2012] WARNING Sector repair completed: port=0, LBA=0xE879057<br />
c0 [Sat Oct 20 00:01:04 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 20 02:19:25 2012] WARNING Sector repair completed: port=1, LBA=0x174B012<br />
c0 [Sat Oct 20 03:41:38 2012] WARNING Sector repair completed: port=0, LBA=0x256D93B<br />
c0 [Sat Oct 20 03:42:01 2012] WARNING Sector repair completed: port=0, LBA=0x256D93D<br />
c0 [Sat Oct 20 03:42:40 2012] WARNING Sector repair completed: port=0, LBA=0x256D940<br />
c0 [Sat Oct 20 03:42:59 2012] WARNING Sector repair completed: port=0, LBA=0x256D942<br />
c0 [Sat Oct 20 03:43:12 2012] WARNING Sector repair completed: port=0, LBA=0x256D944<br />
c0 [Sat Oct 20 03:43:26 2012] WARNING Sector repair completed: port=0, LBA=0x256D948<br />
c0 [Sat Oct 20 16:37:52 2012] WARNING Sector repair completed: port=0, LBA=0xE879060<br />
c0 [Sat Oct 20 16:38:15 2012] WARNING Sector repair completed: port=0, LBA=0xE879062<br />
c0 [Sat Oct 20 21:00:18 2012] INFO Verify completed: unit=0<br />
c0 [Sat Oct 20 23:49:01 2012] WARNING Sector repair completed: port=1, LBA=0x4473E908<br />
c0 [Sun Oct 21 03:42:26 2012] WARNING Sector repair completed: port=1, LBA=0x175BADD5<br />
c0 [Tue Oct 23 01:09:04 2012] WARNING Sector repair completed: port=1, LBA=0x6E524860<br />
c0 [Fri Oct 26 03:21:25 2012] WARNING Sector repair completed: port=0, LBA=0x802C61<br />
c0 [Fri Oct 26 04:22:21 2012] WARNING Sector repair completed: port=0, LBA=0x176353CD<br />
c0 [Sat Oct 27 00:01:03 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 27 00:49:35 2012] WARNING Sector repair completed: port=0, LBA=0x802C65<br />
c0 [Sat Oct 27 17:02:24 2012] WARNING Sector repair completed: port=1, LBA=0xC1FF26D<br />
c0 [Sat Oct 27 17:09:06 2012] WARNING Sector repair completed: port=0, LBA=0xDF621AD<br />
c0 [Sat Oct 27 21:30:57 2012] INFO Verify completed: unit=0<br />
c0 [Tue Oct 30 00:20:46 2012] WARNING Sector repair completed: port=0, LBA=0xE9FE2AB<br />
c0 [Wed Oct 31 02:02:03 2012] WARNING Sector repair completed: port=0, LBA=0x1460C25<br />
c0 [Wed Oct 31 02:04:05 2012] WARNING Sector repair completed: port=0, LBA=0x1460C28<br />
c0 [Thu Nov 1 00:48:34 2012] WARNING Sector repair completed: port=1, LBA=0xA7C92BE<br />
c0 [Thu Nov 1 05:04:45 2012] WARNING Sector repair completed: port=0, LBA=0x1C252C2<br />
<br />
[root@backup1 /data/deprecated]#<br />
</pre><br />
<br />
Look for failed drives and errors. Obviously from the above we need to probably replace drives 0 and 1 and drive 1 is even showing as having problems, yet the RAID array is healthy, amazingly. You also see the automatic verifies.<br />
<br />
Note: when rebuilding a degraded mirror, you will see no progress as it rebuilds in the cli<br />
<pre>Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u0 RAID-5 REBUILDING 0 - 64K 4656.56 OFF ON</pre><br />
<br />
<br />
=== areca ===<br />
<br />
We are using an areca controller on backup3. <br />
<br />
<pre><br />
[root@newbackup3 ~]# sh /root/verify.sh<br />
# Name Raid Name Level Capacity Ch/Id/Lun State<br />
===============================================================================<br />
1 ARC-1160-VOL#00 Raid Set # 00 Raid5 5000.0GB 00/00/00 Checking(19.7%)<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
# Name Disks TotalCap FreeCap DiskChannels State<br />
===============================================================================<br />
1 Raid Set # 00 6 6000.0GB 0.0GB 123456 Checking<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
Date-Time Device Event Type Elapsed Time Errors<br />
===============================================================================<br />
2012-12-05 20:40:58 ARC-1160-VOL#00 Start Checking<br />
2012-12-01 05:06:04 ARC-1160-VOL#00 Complete Init 027:30:45<br />
2012-11-30 01:35:19 ARC-1160-VOL#00 Start Initialize<br />
2026-08-06 01:34:52 H/W MONITOR Raid Powered On<br />
2012-11-30 01:33:36 ARC-1160-VOL#00 Stop Initialization 000:31:48<br />
2012-11-30 01:01:47 ARC-1160-VOL#00 Start Initialize<br />
2026-08-06 00:58:13 H/W MONITOR Raid Powered On<br />
2012-11-30 00:57:26 ARC-1160-VOL#00 Stop Initialization 000:57:07<br />
2012-11-30 00:00:19 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 23:56:48 H/W MONITOR Raid Powered On<br />
2026-08-05 23:52:58 H/W MONITOR Raid Powered On<br />
2026-08-05 23:50:14 H/W MONITOR Raid Powered On<br />
2026-08-05 23:43:30 H/W MONITOR Raid Powered On<br />
2012-11-29 23:10:07 ARC-1160-VOL#00 Stop Initialization 000:00:56<br />
2012-11-29 23:09:11 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 23:08:57 H/W MONITOR Raid Powered On<br />
2012-11-29 23:08:10 ARC-1160-VOL#00 Stop Initialization 000:20:41<br />
2012-11-29 22:47:29 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 22:46:59 H/W MONITOR Raid Powered On<br />
2026-08-05 22:45:55 H/W MONITOR Raid Powered On<br />
2026-08-05 22:44:53 H/W MONITOR Raid Powered On<br />
2026-08-05 22:42:06 H/W MONITOR Raid Powered On<br />
2026-08-05 22:40:50 H/W MONITOR Raid Powered On<br />
2012-11-29 22:40:04 ARC-1160-VOL#00 Stop Initialization 000:24:25<br />
2012-11-29 22:15:38 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 22:15:11 000:000001215B00 Restart Init LBA Point<br />
2026-08-05 22:15:10 H/W MONITOR Raid Powered On<br />
2012-11-29 21:56:38 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 21:56:12 H/W MONITOR Raid Powered On<br />
2026-08-05 21:56:04 IDE Channel #03 Device Inserted<br />
2012-11-29 21:55:13 IDE Channel #04 Device Inserted<br />
2012-11-29 21:55:03 IDE Channel #02 Device Inserted<br />
2026-08-05 21:53:09 H/W MONITOR Raid Powered On<br />
2026-08-05 20:51:46 H/W MONITOR Raid Powered On<br />
2026-08-05 20:49:56 H/W MONITOR Raid Powered On<br />
2026-08-05 20:48:29 H/W MONITOR Raid Powered On<br />
2026-08-05 20:46:29 H/W MONITOR Raid Powered On<br />
2026-08-05 20:44:49 H/W MONITOR Raid Powered On<br />
2026-08-05 20:43:01 H/W MONITOR Raid Powered On<br />
2026-08-05 20:36:25 H/W MONITOR Raid Powered On<br />
2026-08-05 20:31:18 H/W MONITOR Raid Powered On<br />
2026-08-05 20:30:08 H/W MONITOR Raid Powered On<br />
2026-08-05 20:08:40 H/W MONITOR Raid Powered On<br />
2026-08-05 20:06:11 H/W MONITOR Raid Powered On<br />
2026-08-05 20:05:14 H/W MONITOR Raid Powered On<br />
2026-08-05 20:03:58 H/W MONITOR Raid Powered On<br />
2026-08-05 20:00:56 H/W MONITOR Raid Powered On<br />
2026-08-05 19:57:57 H/W MONITOR Raid Powered On<br />
2026-08-05 19:56:15 H/W MONITOR Raid Powered On<br />
2026-08-05 19:55:05 H/W MONITOR Raid Powered On<br />
2026-08-05 17:24:36 H/W MONITOR Raid Powered On<br />
2026-08-05 17:22:43 H/W MONITOR Raid Powered On<br />
2026-08-05 04:50:42 H/W MONITOR Raid Powered On<br />
2026-08-05 04:47:33 H/W MONITOR Raid Powered On<br />
2026-08-05 04:43:57 H/W MONITOR Raid Powered On<br />
2026-08-05 04:18:52 H/W MONITOR Raid Powered On<br />
2026-08-05 04:17:30 H/W MONITOR Raid Powered On<br />
2026-08-05 04:13:30 H/W MONITOR Raid Powered On<br />
2026-08-05 04:10:26 H/W MONITOR Raid Powered On<br />
2026-08-05 04:09:23 H/W MONITOR Raid Powered On<br />
2026-08-05 00:08:09 H/W MONITOR Raid Powered On<br />
2026-08-05 00:07:12 H/W MONITOR Raid Powered On<br />
2026-08-05 00:05:51 H/W MONITOR Raid Powered On<br />
2026-08-05 00:04:27 H/W MONITOR Raid Powered On<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
press enter when ready to run verify<br />
</pre><br />
<br />
Look for failed drives and errors. <br />
<br />
When it proceed's to verifying, you can confirm with:<br />
<br />
<pre><br />
[root@newbackup3 ~]# cli64 vsf info<br />
# Name Raid Name Level Capacity Ch/Id/Lun State<br />
===============================================================================<br />
1 ARC-1160-VOL#00 Raid Set # 00 Raid5 5000.0GB 00/00/00 Checking(22.5%)<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
[root@newbackup3 ~]#<br />
</pre><br />
<br />
== Update OS list ==<br />
<br />
# check for any new VZ templates we want to offer: <tt>vzup2date -z</tt><br />
# see if there's any OS's we want to include in our colo install list. Update 2 places: <tt>signup/html/colo_quote.html</tt> & <tt>signup/html/step1.html</tt><br />
# update the mgmt database (ref_templates table, ref_systems table).<br />
<br />
= Infrequent tasks =<br />
<br />
== Free up space on gateway ==<br />
<br />
<pre>newgateway /var/spool# cd clientmqueue/<br />
newgateway /var/spool/clientmqueue# sh<br />
# for f in `ls`; do rm $f; done<br />
exit</pre><br />
<br />
== Free up space on mail ==<br />
<br />
You can clear out root mail:<br />
<pre><br />
mail /var/log# ll -h /var/mail/root<br />
-rw------- 1 root mail 543K Dec 19 13:05 /var/mail/root<br />
mail /var/log# rm /var/mail/root<br />
</pre><br />
<br />
Or you can archive mail logs:<br />
<pre><br />
mail /var/log# ls -l htt*<br />
-rw-r--r-- 1 root wheel 297436931 Dec 19 13:26 httpd-access.log<br />
-rw-r--r-- 1 root wheel 9824324 Jul 4 11:34 httpd-access.log.old.0.gz<br />
-rw-r--r-- 1 root wheel 6884137 Mar 17 2012 httpd-access.log.old.1.gz<br />
-rw-r--r-- 1 root wheel 18557444 Dec 3 2009 httpd-access.log.old.10.gz<br />
-rw-r--r-- 1 root wheel 14740263 Jan 9 2007 httpd-access.log.old.11.gz<br />
-rw-r--r-- 1 root wheel 14209465 Nov 28 2007 httpd-access.log.old.12.gz<br />
-rw-r--r-- 1 root wheel 16874396 Feb 19 2012 httpd-access.log.old.3.gz<br />
-rw-r--r-- 1 root wheel 14554859 Jul 22 2011 httpd-access.log.old.4.gz<br />
-rw-r--r-- 1 root wheel 10513227 Feb 18 2011 httpd-access.log.old.5.gz<br />
-rw-r--r-- 1 root wheel 7201946 Oct 29 2010 httpd-access.log.old.6.gz<br />
-rw-r--r-- 1 root wheel 10062537 May 6 2010 httpd-access.log.old.7.gz<br />
-rw-r--r-- 1 root wheel 10157042 Aug 12 2010 httpd-access.log.old.8.gz<br />
-rw-r--r-- 1 root wheel 11909534 Mar 4 2010 httpd-access.log.old.9.gz<br />
-rw-r--r-- 1 root wheel 59030930 Dec 19 13:01 httpd-error.log<br />
-rw-r--r-- 1 root wheel 3413134 Mar 4 2010 httpd-error.log.0.gz<br />
-rw-r--r-- 1 root wheel 795515 May 1 2007 httpd-error.log.1.gz<br />
-rw-r--r-- 1 root wheel 1142153 Nov 30 2007 httpd-error.log.2.gz<br />
-rw-r--r-- 1 root wheel 2325801 Feb 18 2011 httpd-error.log.gz<br />
<br />
mail /var/log# sh<br />
# for f in 12 11 10 9 8 7 6 5 4 3 2 1 0; do g=`echo $f+1|bc`; mv httpd-access.log.old.$f.gz httpd-access.log.old.$g.gz; done<br />
# mv httpd-access.log httpd-access.log.old.0<br />
# touch httpd-access.log<br />
# apachectl restart<br />
# gzip httpd-access.log.old.0<br />
<br />
# for f in 2 1 0; do g=`echo $f+1|bc`; mv httpd-error.log.$f.gz httpd-error.log.$g.gz; done<br />
# mv httpd-error.log httpd-error.log.0<br />
# touch httpd-error.log<br />
# apachectl restart<br />
# gzip httpd-error.log.0<br />
# exit<br />
</pre><br />
<br />
== Free up space on bwdb2 ==<br />
<br />
You can either remove items from <tt>/usr/home/archive</tt> or you can scp them to backup3:/data/bwdb2/archive .<br />
<br />
== Free up space on backup1 ==<br />
<br />
backup1 is our primary customer backup system. As usage grows over time, it needs to be regularly purged of old files. The easiest way to do this is by removing deprecated files. These mostly consist of cancelled customers or temporary dump/storage files (created during dump/restores). Our standard policy is to hang onto cancelled customers for 6mos after which we remove their files (as far as customers know their data is purged immediately, but we hang onto it just in case.. and in some cases we cancel a server due to non payment so this makes it easy to restore their system). To find files to remove:<br />
<br />
<pre>[root@backup1 ~]# cd /data/deprecated/<br />
[root@backup1 /data/deprecated]# ls<br />
2101-migrated-20120317.tgz old-683-cxld-20121021.tgz<br />
69.55.230.2-wwwbackup old-744-cxld-20120708.tgz<br />
991-DONTDELETE.tgz old-809-cxld-20120609.tgz<br />
archive-col02050-mdfile-cxld-20120409.gz old-854-cxld-20120621.tgz<br />
col01371.tgz old-931-cxld-20060513.tgz<br />
deleteme_ubuntu-10.10-x86_20111205 old-col00123-mdfile-noarchive-20120417.gz<br />
jail10_old old-col00147-vnfile-cxld-20120828.gz<br />
jail14_rsync_old old-col00419-dump-cxld-20120224.gz<br />
jail15_old old-col01098-vnfile-cxld-20120827.gz<br />
jail3_old old-col01278-dump-cxld-20120822<br />
jail4_old old-col01517-dump-cxld-20120828<br />
jail5_old old-col01669-dump-cxld-20120203.gz<br />
old-1009-cxld-20120608.tgz old-col01687-dump-cxld-20120909<br />
old-1012-cxld-20120411.tgz old-col01790-dump-cxld-20120828<br />
old-1052-cxld-20120721.tgz old-col01812-dump-cxld-20120820<br />
old-10631-cxld-20120622.tgz old-col01938-mdfile-cxld-20120619.gz<br />
old-10632-cxld-20120622.tgz old-col02095-mdfile-noarchive-20120523.gz<br />
old-10633-cxld-20120622.tgz olddebian-3.0-v15-20110610.tgz<br />
old-1236-cxld-20120621.tgz oldmod_frontpage-deb30-v15-20110610.tgz<br />
old-1381-cxld-20120404.tgz oldmod_perl-deb30-v15-20110610.tgz<br />
old-1422-cxld-20120721.tgz oldmod_ssl-deb30-v15-20110610.tgz<br />
old-14681-cxld-20120619.tgz oldmysql-deb30-v15-20110610.tgz<br />
old-1544-cxld-20120626.tgz oldproftpd-deb30-v15-20110610.tgz<br />
old-18351-cxld-20120605.tgz old_virt14<br />
old-1853-cxld-20120910.tgz old_virt18<br />
old-1963-cxld-20120206.tgz oldwebmin-deb30-v15-20110610.tgz<br />
old-1967-cxld-20120605.tgz suse.virt11.20120421.tgz<br />
old-1981-noarchive-20120729.tgz virt11<br />
old-2030-migrated-noarchive-20120727.tgz virt12_old<br />
old-2037-cxld-20120716.tgz virt13_old<br />
old-2065-cxld-20120727.tgz virt16_old<br />
old-2068-cxld-20120424.tgz virt4_old<br />
old-2085-cxld-20120531.tgz virt5_old<br />
old-364-cxld-20120904.tgz virt6_old<br />
old-446-cxld-20120512.tgz virt7_old<br />
old-613-cxld-20120601.tgz virt8_old<br />
[root@backup1 /data/deprecated]#<br />
</pre><br />
<br />
virtX_old and jailX_old are permanently archived, so ignore those as well as anything else marked not to delete or otherwise suspicious. Likewise, probably a good idea to try to hang onto oldTEMPLATE.gz as long as we can as well.<br />
Most of the stuff we want to delete is dated when it was deprecated, making this easy. So to remove files from 6 mos ago (running this in Oct):<br />
<br />
<pre>[root@backup1 /data/deprecated]# ls old*201204*<br />
old-1012-cxld-20120411.tgz old-2068-cxld-20120424.tgz<br />
old-1381-cxld-20120404.tgz old-col00123-mdfile-noarchive-20120417.gz<br />
[root@backup1 /data/deprecated]# rm old*201204*<br />
</pre><br />
<br />
Every few months you will also want to remove some of the snapshot archives for mail. We typically save the 1st, 10th, and 20th of each month. To do this you set aside the dates you want to save then remove months at a time, followed by restoring the set aside dates. Here's how that works:<br />
<pre>[root@backup1 /data/www/daily]# ls<br />
05 08-10-11 10-04-10 11-10-10 12-07-29 12-09-21 12-11-14<br />
06 08-10-21 10-04-20 11-10-20 12-07-30 12-09-22 12-11-15<br />
06-06-01-usr-home.tgz 08-11-01 10-05-01 11-11-01 12-07-31 12-09-23 12-11-16<br />
06-07-01-usr-home.tgz 08-11-10 10-05-11 11-11-10 12-08-01 12-09-24 12-11-17<br />
06-08-01-usr-home.tgz 08-11-20 10-05-20 11-11-20 12-08-02 12-09-25 12-11-18<br />
06-09-01-usr-home.tgz 08-12-01 10-06-01 11-12-01 12-08-03 12-09-26 12-11-19<br />
06-11-10 08-12-10 10-06-10 11-12-10 12-08-04 12-09-27 12-11-20<br />
06-12-21 08-12-20 10-06-20 11-12-20 12-08-05 12-09-28 12-11-21<br />
07-01-10 09-01-01 10-07-01 12-01-01 12-08-06 12-09-29 12-11-22<br />
07-01-20 09-01-10 10-07-10 12-01-10 12-08-07 12-09-30 12-11-23<br />
07-02-10 09-01-20 10-07-20 12-01-20 12-08-08 12-10-01 12-11-24<br />
07-02-20 09-02-01 10-08-01 12-02-01 12-08-09 12-10-02 12-11-25<br />
07-03-01 09-02-10 10-08-10 12-02-10 12-08-10 12-10-03 12-11-26<br />
07-03-20 09-02-20 10-08-20 12-02-20 12-08-11 12-10-04 12-11-27<br />
07-04-01 09-03-01 10-09-01 12-03-01 12-08-12 12-10-05 12-11-28<br />
07-04-10 09-03-10 10-09-10 12-03-10 12-08-13 12-10-06 12-11-29<br />
07-04-20 09-03-20 10-09-20 12-03-20 12-08-14 12-10-07 12-11-30<br />
07-05-01 09-04-01 10-10-01 12-04-01 12-08-15 12-10-08 12-12-01<br />
07-05-10 09-04-10 10-10-10 12-04-10 12-08-16 12-10-09 12-12-02<br />
07-05-20 09-04-20 10-10-20 12-04-20 12-08-17 12-10-10 12-12-03<br />
07-06-01 09-05-01 10-11-01 12-05-01 12-08-18 12-10-11 12-12-04<br />
07-06-10 09-05-10 10-11-10 12-05-10 12-08-19 12-10-12 12-12-05<br />
07-06-20 09-05-20 10-11-20 12-05-20 12-08-20 12-10-13 12-12-06<br />
07-07-20 09-06-01 10-12-01 12-06-01 12-08-21 12-10-14 12-12-07<br />
07-08-10 09-06-10 10-12-10 12-06-10 12-08-22 12-10-15 12-12-08<br />
07-08-20 09-06-20 10-12-20 12-06-20 12-08-23 12-10-16 12-12-09<br />
07-09-01 09-07-01 11-01-01 12-07-01 12-08-24 12-10-17 12-12-10<br />
07-10-01 09-07-10 11-01-10 12-07-02 12-08-25 12-10-18 12-12-11<br />
07-10-10 09-07-20 11-01-21 12-07-03 12-08-26 12-10-19 12-12-12<br />
07-10-20 09-08-01 11-02-01 12-07-04 12-08-27 12-10-20 12-12-13<br />
07-12-01 09-08-10 11-02-10 12-07-05 12-08-28 12-10-21 12-12-14<br />
07-12-10 09-08-20 11-02-20 12-07-06 12-08-29 12-10-22 12-12-15<br />
08-01-01 09-09-01 11-03-01 12-07-07 12-08-30 12-10-23 12-12-16<br />
08-01-20 09-09-10 11-03-10 12-07-08 12-08-31 12-10-24 12-12-17<br />
08-02-20 09-09-20 11-03-20 12-07-09 12-09-01 12-10-25 12-12-18<br />
08-03-01 09-10-01 11-04-01 12-07-10 12-09-02 12-10-26 12-12-19<br />
08-03-10 09-10-10 11-04-10 12-07-11 12-09-03 12-10-27 12-12-20<br />
08-03-20 09-10-20 11-04-20 12-07-12 12-09-04 12-10-28 12-12-21<br />
08-04-01 09-11-01 11-05-01 12-07-13 12-09-05 12-10-29 12-12-22<br />
08-04-20 09-11-10 11-05-10 12-07-14 12-09-06 12-10-30 12-12-23<br />
08-05-01 09-11-20 11-05-20 12-07-15 12-09-07 12-10-31 12-12-24<br />
08-05-10 09-12-01 11-06-01 12-07-16 12-09-08 12-11-01 12-12-25<br />
08-06-10 09-12-10 11-06-10 12-07-17 12-09-09 12-11-02 12-12-26<br />
08-06-20 09-12-20 11-06-20 12-07-18 12-09-10 12-11-03 12-12-27<br />
08-07-02 10-01-01 11-07-01 12-07-19 12-09-11 12-11-04 12-12-28<br />
08-07-10 10-01-10 11-07-10 12-07-20 12-09-12 12-11-05 2008-10-23<br />
08-07-20 10-01-20 11-07-20 12-07-21 12-09-13 12-11-06 bb.tgz<br />
08-08-01 10-02-01 11-08-01 12-07-22 12-09-14 12-11-07 boot<br />
08-08-10 10-02-10 11-08-10 12-07-23 12-09-15 12-11-08 current<br />
08-08-21 10-02-20 11-08-20 12-07-24 12-09-16 12-11-09 hold<br />
08-09-01 10-03-01 11-09-01 12-07-25 12-09-17 12-11-10<br />
08-09-10 10-03-10 11-09-10 12-07-26 12-09-18 12-11-11<br />
08-09-21 10-03-20 11-09-20 12-07-27 12-09-19 12-11-12<br />
08-10-01 10-04-01 11-10-01 12-07-28 12-09-20 12-11-13<br />
[root@backup1 /data/www/daily]#<br />
</pre><br />
<br />
So we see that everything up to July 2012 has been pruned. To prune July 2012 we do the following:<br />
<br />
<pre>mv 12-07-01 hold<br />
mv 12-07-10 hold<br />
mv 12-07-20 hold<br />
rm -fr 12-07*<br />
mv hold/* .</pre></div>75.80.128.147https://wiki.jcihosting.com/index.php?title=Routine_Maintenance&diff=2202Routine Maintenance2020-02-25T17:53:27Z<p>75.80.128.147: /* Nagios */</p>
<hr />
<div>= Daily Tasks =<br />
<br />
== check load graphs == <br />
<br />
Click on the Load link in mgmt <br />
<br />
This screen shows you load levels on our servers and network traffic for critical machines (firewalls, backup servers).<br />
<br />
If you see load high or increasing <br />
<br />
FreeBSD: <br />
run [[VPS_Management#jtop|jtop]] (or [[VPS_Management#jt|jt]] > 7.x) and see if there are any runaway processes. Here are some examples of entries in top that are<br />
definitely runaway processes:<br />
<br />
<pre>79481 root 64 0 2256K 1056K CPU1 1 58:16 87.40% 87.40% nano<br />
50650 1000 64 0 1852K 1112K RUN 0 207.9H 84.08% 84.08% screen<br />
14829 www 2 0 39100K 31736K accept 0 104:24 46.54% 6.54% httpd<br />
42065 root 61 0 1300K 844K RUN 1 47.8H 91.36% 91.36% ee<br />
1328 www 56 0 18440K 10796K CPU1 0 64.4H 97.71% 97.71% httpd<br />
26251 user 57 0 6124K 1160K CPU1 1 82.9H 98.44% 98.44% screen<br />
89874 root 60 0 1352K 892K RUN 1 33.8H 65.82% 65.82% dialog<br />
38656 1000 64 0 3088K 2136K CPU0 0 806:13 97.95% 97.95% StutBot<br />
27630 root 64 0 1396K 972K RUN 1 76.8H 86.47% 86.47% ee</pre><br />
<br />
Linux:<br />
run [[VPS_Management#vwe|vwe]] to see which VPS’s have high loads. From there run <tt>[[VPS_Management#vp|vp]] <veid></tt> and/or <tt>[[VPS_Management#vt|vt]] <veid></tt> to see what's going on in that system.<br />
[[VPS_Management#vzstat|vzstat]] will also give you a nice picture of whats going on, systems with high numbers in the mlat column are likely culprits.<br />
<br />
examples of out of control procs:<br />
<br />
<pre>12183 nobody 16 0 4916 1348 1340 R 45.5 0.0 4249m httpd<br />
29266 #502 16 0 1852 796 792 R 22.5 0.0 1104m vim<br />
23860 #41 16 0 5472 5472 2076 R 98.9 0.2 31:41 python<br />
19227 bin 19 0 1688 716 652 R 99.9 0.0 321:08 wtrs_ui<br />
7762 apache 16 0 268 236 224 R 85.7 0.0 1010m ptrace<br />
4624 #501 20 0 4304 2400 2044 R 53.6 0.1 284:32 YoSucker<br />
20451 #506 20 0 1876 820 816 R 17.2 0.0 169:35 vim<br />
8834 #514 20 0 900 724 672 R 77.6 0.0 382:30 neostats<br />
31815 apache 14 0 3176 3176 1696 R 74.4 0.1 6:15 counter</pre><br />
<br />
Just kill -9 them and be done with it.<br />
<br />
Also, anytime you see `kmod` or `ptrace` - kill those immediaely no matter how much they are using - they are attempts to exploit the linux ptrace bug. They won't work, but they suck a lot of CPU...<br />
<br />
Also, any other processes that are at 90-100% cpu usage and have been running for any long period of time should be killed except for mysqld processes on FreeBSD. See above.<br />
<br />
However, there is an exception:<br />
<br />
if it is a mysqld, we don't want to kill their database. What you want to do is <tt>[[VPS_Management#jpid|jpid]] <pid></tt> to see who owns it, and then email them the paste containing the instructions for the nanny. Or you can simply do a <tt>kill -1 PID</tt> on the process to restart it.<br />
<br />
=== Load averages jump at night ===<br />
<br />
The load averages on the FreeBSD systems may jump up at night between 1 and 4 am - this is because the backups are running - if this is what is causing a jump in load, you will see processes like `rsync` in top eating a lot of CPU time.<br />
<br />
== check backups ==<br />
<br />
mgmt -> Motnroing -> Backups and make sure every machine was backed up the previous nite. <br />
Also look at df on backup1 and backup2 to make sure no disk is approaching full, though bb should warn us in advance. Please note - errors encountered when a backup script on any of the particular systems run will gnerate an email to support@johncompanies.com so you can know immediately the day after if the directory to be backed up has been moved or no longer exists. A paste exists for this to notify the customer of a non-existant file/dir.<br />
<br />
== check bb for warnings ==<br />
mgmt -> BigBrother<br />
<br />
Some events don't generate pages (on purpose). You will only see them by going to the bb main page.<br />
<br />
== check jail5 for crashed VPSs ==<br />
<br />
On jail5<br />
<br />
notrunning<br />
<br />
To restart a VPS <br />
<br />
vm restart col0XXXX<br />
<br />
== check NetHere ==<br />
<br />
Check the NetHere servers<br />
<br />
=== Mail systems ===<br />
Check for possible SPAMMERs.<br />
<br />
==== Incoming ====<br />
Check mta-1 and mta-2 count of customer logins for possible<br />
customer SPAM compromises.<br />
<br />
login_count /logs/maillog | tail -30<br />
<br />
==== Outgoing ====<br />
<br />
Check outgoing queues on relay-1 and relay-2<br />
<br />
mail_count | tail -30<br />
<br />
=== Nagios ===<br />
<br />
Check for other problems on NetHere servers<br />
<br />
https://nagios.nethere.net<br />
<br />
=== Cacti ===<br />
<br />
Check bandwidth usage on servers<br />
<br />
https://cacti.nethere.net<br />
<br />
= Monthly Tasks =<br />
== rotate pine sent mail (1st of month) ==<br />
On the 1st of the month, before any emails are sent out, quit out of pine, then log back in. Send mail from last month will be archived.<br />
If you mess up and do it on the 3rd (for example), you can go into the previous month's saved email and save emails from the current month into the <tt>sent-mail</tt> (current month) mailbox.<br />
<br />
== b/w caps ==<br />
On the 1st: remove any bwcaps put into the firewall (only really applies if a bwcap was added cause someone went over on b/w):<br />
<br />
ipfw list|grep pipe<br />
ipfw del [each rule listed]<br />
<br />
NOTE: this cronjob on newgateway will do some of that for you, provided you used one of the following pipe #s:<br />
0 0 1 * * /sbin/ipfw del 3 4 5 17331<br />
<br />
-------------------<br />
<br />
We really don’t do this anymore since we have centralized traffic accounting with netflow, but for posterity:<br />
<br />
Make sure all machines reset counters to 0 after midnight on the 1st<br />
Make sure they dumped a counter<br />
<br />
On each jail run:<br />
trafficgather.pl<br />
<br />
And on each virt:<br />
linuxtrafficgather.pl<br />
<br />
== Monthly RAID checks ==<br />
<br />
Every month we check the health of and verfy the parity on all our RAID-based systems.<br />
To facilitate this, we've created a simple script to start the process:<br />
<br />
sh /root/verify.sh<br />
<br />
=== Adaptec controllers ===<br />
Here's some sample output:<br />
<pre><br />
mail /usr/local/www/scripts# sh /root/verify.sh<br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> container list /f<br />
Executing: container list /full=TRUE<br />
Num Total Oth Chunk Scsi Partition<br />
Creation System<br />
Label Type Size Ctr Size Usage B:ID:L Offset:Size State RO Lk Task Done% Ent<br />
Date Time Files<br />
----- ------ ------ --- ------ ------- ------ ------------- ------- -- -- ------- ------ ---<br />
------ -------- ------<br />
0 Mirror 33.9GB Open 0:01:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:32<br />
/dev/aacd0 mirror0 0:00:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:32<br />
<br />
1 Mirror 33.9GB Open 0:02:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:50<br />
/dev/aacd1 mirror1 0:03:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:50<br />
<br />
<br />
AAC0> disk list /f<br />
Executing: disk list /full=TRUE<br />
<br />
B:ID:L Device Type Removable media Vendor-ID Product-ID Rev Blocks Bytes/Bl<br />
ock Usage Shared Rate<br />
------ -------------- --------------- --------- ---------------- ----- --------- --------<br />
--- ---------------- ------ ----<br />
0:00:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:01:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:02:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:03:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
<br />
AAC0> disk show smart<br />
Executing: disk show smart<br />
<br />
Smart Method of Enable<br />
Capable Informational Exception Performance Error<br />
B:ID:L Device Exceptions(MRIE) Control Enabled Count<br />
------ ------- ---------------- --------- ----------- ------<br />
0:00:0 Y 6 Y N 0<br />
0:01:0 Y 6 Y N 0<br />
0:02:0 Y 6 Y N 0<br />
0:03:0 Y 6 Y N 0<br />
0:06:0 N<br />
<br />
AAC0> task list<br />
Executing: task list<br />
<br />
Controller Tasks<br />
<br />
TaskId Function Done% Container State Specific1 Specific2<br />
------ -------- ------- --------- ----- --------- ---------<br />
<br />
No tasks currently running on controller<br />
<br />
AAC0> dia sh hi<br />
Executing: diagnostic show history<br />
No switches specified, defaulting to "/current".<br />
<br />
<br />
<br />
*** HISTORY BUFFER FROM CURRENT CONTROLLER RUN ***<br />
<br />
[00]: GetDiskLogEntry: container - 1, entry return 0<br />
[01]: Container 1 started SCRUB task<br />
[02]: Starting Mirror:1 scrub<br />
[03]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[04]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[05]: UpdateDiskLogIndex - Set - container 0, index 1<br />
[06]: GetDiskLogEntry: container - 0, entry return 1<br />
[07]: Container 0 started SCRUB task<br />
[08]: Starting Mirror:0 scrub<br />
[09]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[10]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[11]: Mirror Scrub Container:1 ErrorsFound:0<br />
[12]: Clear disk log: sector - 80, driveno 2<br />
[13]: Clear disk log: sector - 80, driveno 3<br />
[14]: Container 1 completed SCRUB task:<br />
[15]: Mirror Scrub Container:0 ErrorsFound:0<br />
[16]: Clear disk log: sector - 81, driveno 1<br />
[17]: Clear disk log: sector - 81, driveno 0<br />
[18]: Container 0 completed SCRUB task:<br />
[19]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[20]: GetDiskLogEntry: container - 0, entry return 0<br />
[21]: Container 0 started SCRUB task<br />
[22]: Starting Mirror:0 scrub<br />
[23]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[24]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[25]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[26]: GetDiskLogEntry: container - 1, entry return 1<br />
[27]: Container 1 started SCRUB task<br />
[28]: Starting Mirror:1 scrub<br />
[29]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[30]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[31]: Mirror Scrub Container:1 ErrorsFound:0<br />
[32]: Clear disk log: sector - 81, driveno 2<br />
[33]: Clear disk log: sector - 81, driveno 3<br />
[34]: Container 1 completed SCRUB task:<br />
[35]: Mirror Scrub Container:0 ErrorsFound:0<br />
[36]: Clear disk log: sector - 80, driveno 1<br />
[37]: Clear disk log: sector - 80, driveno 0<br />
[38]: Container 0 completed SCRUB task:<br />
[39]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[40]: GetDiskLogEntry: container - 0, entry return 0<br />
[41]: Container 0 started SCRUB task<br />
[42]: Starting Mirror:0 scrub<br />
[43]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[44]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[45]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[46]: GetDiskLogEntry: container - 1, entry return 1<br />
[47]: Container 1 started SCRUB task<br />
[48]: Starting Mirror:1 scrub<br />
[49]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[50]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[51]: Mirror Scrub Container:1 ErrorsFound:0<br />
[52]: Clear disk log: sector - 81, driveno 2<br />
[53]: Clear disk log: sector - 81, driveno 3<br />
[54]: Container 1 completed SCRUB task:<br />
[55]: Mirror Scrub Container:0 ErrorsFound:0<br />
[56]: Clear disk log: sector - 80, driveno 1<br />
[57]: Clear disk log: sector - 80, driveno 0<br />
[58]: Container 0 completed SCRUB task:<br />
[59]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[60]: GetDiskLogEntry: container - 0, entry return 0<br />
[61]: Container 0 started SCRUB task<br />
[62]: Starting Mirror:0 scrub<br />
[63]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[64]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[65]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[66]: GetDiskLogEntry: container - 1, entry return 1<br />
[67]: Container 1 started SCRUB task<br />
[68]: Starting Mirror:1 scrub<br />
[69]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[70]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[71]: Mirror Scrub Container:1 ErrorsFound:0<br />
[72]: Clear disk log: sector - 81, driveno 2<br />
[73]: Clear disk log: sector - 81, driveno 3<br />
[74]: Container 1 completed SCRUB task:<br />
[75]: Mirror Scrub Container:0 ErrorsFound:0<br />
[76]: Clear disk log: sector - 80, driveno 1<br />
[77]: Clear disk log: sector - 80, driveno 0<br />
[78]: Container 0 completed SCRUB task:<br />
[79]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[80]: GetDiskLogEntry: container - 0, entry return 0<br />
[81]: Container 0 started SCRUB task<br />
[82]: Starting Mirror:0 scrub<br />
[83]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[84]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[85]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[86]: GetDiskLogEntry: container - 1, entry return 1<br />
[87]: Container 1 started SCRUB task<br />
[88]: Starting Mirror:1 scrub<br />
[89]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[90]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[91]: Mirror Scrub Container:1 ErrorsFound:0<br />
[92]: Clear disk log: sector - 81, driveno 2<br />
[93]: Clear disk log: sector - 81, driveno 3<br />
[94]: Container 1 completed SCRUB task:<br />
[95]: Mirror Scrub Container:0 ErrorsFound:0<br />
[96]: Clear disk log: sector - 80, driveno 1<br />
[97]: Clear disk log: sector - 80, driveno 0<br />
[98]: Container 0 completed SCRUB task:<br />
[99]:<br />
<br />
========================<br />
History Output Complete.<br />
<br />
AAC0><br />
AAC0> exit<br />
Executing: exit<br />
<br />
press enter when ready to run verify <INS><br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> contai scr 0<br />
Executing: container scrub 0<br />
<br />
AAC0> contai scr 1<br />
Executing: container scrub 1<br />
<br />
AAC0> exit<br />
Executing: exit<br />
<br />
when done run: <br />
<br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
<br />
<br />
Nov 1 10:32:46 mail /kernel: aac0: **Monitor** Container 0 started SCRUB task<br />
Nov 1 10:32:47 mail /kernel: aac0: **Monitor** Container 1 started SCRUB task<br />
<br />
</pre><br />
<br />
Here's an analysis of what we're seeing and what we're looking for:<br />
<br />
<pre><br />
AAC0> container list /f<br />
Executing: container list /full=TRUE<br />
Num Total Oth Chunk Scsi Partition<br />
Creation System<br />
Label Type Size Ctr Size Usage B:ID:L Offset:Size State RO Lk Task Done% Ent<br />
Date Time Files<br />
----- ------ ------ --- ------ ------- ------ ------------- ------- -- -- ------- ------ ---<br />
------ -------- ------<br />
0 Mirror 33.9GB Open 0:01:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:32<br />
/dev/aacd0 mirror0 0:00:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:32<br />
<br />
1 Mirror 33.9GB Open 0:02:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:50<br />
/dev/aacd1 mirror1 0:03:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:50<br />
</pre><br />
<br />
This is showing you the health of the arrays. You're looking for ''Normal'' under the State column, and the absence of a ! in the sector size - sometimes, you'll see this:<br />
64.0KB!33.9GB <br />
<br />
That indicates a problem.<br />
<br />
<pre><br />
AAC0> disk show smart<br />
Executing: disk show smart<br />
<br />
Smart Method of Enable<br />
Capable Informational Exception Performance Error<br />
B:ID:L Device Exceptions(MRIE) Control Enabled Count<br />
------ ------- ---------------- --------- ----------- ------<br />
0:00:0 Y 6 Y N 0<br />
0:01:0 Y 6 Y N 0<br />
0:02:0 Y 6 Y N 0<br />
0:03:0 Y 6 Y N 0<br />
0:06:0 N<br />
</pre><br />
<br />
This shows you a SMART report output. Looking for values in the Error Count column.<br />
<pre><br />
AAC0> task list<br />
Executing: task list<br />
<br />
Controller Tasks<br />
<br />
TaskId Function Done% Container State Specific1 Specific2<br />
------ -------- ------- --------- ----- --------- ---------<br />
<br />
No tasks currently running on controller<br />
</pre><br />
<br />
Look for absence of tasks running- a bad thing would be to see a rebuild or verify running when you didn't initiate it.<br />
<br />
With the history output, you're looking for any anomalies or events since the last time a verify was run. If you see a drive with lots of problems, you may want to take backups before allowing the verify to run since it could replicate errors onto the good drive.<br />
<br />
After you see the history output, it will prompt you to press enter to run the verify. If you're happy with all the output you're seeing- mirror is healthy, history looks good, it's safe to proceed. Otherwise ^C to exit. After hitting enter it will start the verify and start to tail the messages log file (so you can easily see when the verify is complete). Here's what that'll look like:<br />
<br />
<pre><br />
Nov 1 14:38:08 mail /kernel: aac0: **Monitor** Container 1 completed SCRUB task:<br />
Nov 1 14:46:45 mail /kernel: aac0: **Monitor** Container 0 completed SCRUB task:<br />
</pre><br />
<br />
So, putting it all together, after hitting enter to start the verify, you'll see:<br />
<br />
<pre><br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> contai scr 0<br />
Executing: container scrub 0<br />
<br />
AAC0> contai scr 1<br />
Executing: container scrub 1<br />
<br />
AAC0> exit<br />
Executing: exit<br />
<br />
when done run: <br />
<br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
<br />
<br />
Nov 1 10:32:46 mail /kernel: aac0: **Monitor** Container 0 started SCRUB task<br />
Nov 1 10:32:47 mail /kernel: aac0: **Monitor** Container 1 started SCRUB task<br />
<br />
</pre><br />
<br />
When the scrub(s) (verify) are complete - if the server has multiple logical drives, it will run both in parallel - you should exit the tail of the log file (^C) and run:<br />
<br />
<pre><br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
</pre><br />
<br />
Which will show you the diagnostic history, you're looking for the results of the most recent scrub:<br />
<br />
<pre><br />
[100]: Mirror Scrub Container:1 ErrorsFound:0<br />
[101]: Clear disk log: sector - 81, driveno 2<br />
[102]: Clear disk log: sector - 81, driveno 3<br />
[103]: Container 1 completed SCRUB task:<br />
[104]: Mirror Scrub Container:0 ErrorsFound:0<br />
[105]: Clear disk log: sector - 80, driveno 1<br />
[106]: Clear disk log: sector - 80, driveno 0<br />
[107]: Container 0 completed SCRUB task:<br />
</pre><br />
<br />
^C to exit the RAID CLI.<br />
<br />
If you see:<br />
<br />
[104]: Mirror Scrub Container:0 ErrorsFound:5<br />
<br />
You'll want to rerun the verify on that drive till it shows 0, or perhaps replace the drive- you should be able to see from the output which drive had the problem.<br />
<br />
Depending on the size and how busy the drive is, the verify can take anywhere from an hour to the better part of a day.<br />
<br />
You will notice that the diagnostic history is not shown on our modern adaptec cards (i.e. any adaptec card not in a Dell 2450). The reason for this is the history is never cleared, so there's simply too much data to show and it just crashes the CLI. So, don't bother trying to see it...which does make it hard to see if there are problems going on, so you just need to watch the scrub to see it goes to 100%. You will also notice that on some servers there's no tail of messages. Again, this is cause no data is shown there about the completion of the scrub. The thing to do here is to go into the CLI and continue to <tt>show tasks</tt> to monitor scrub progress.<br />
<br />
See [[RAIC_CLI#Adaptec|Adaptec RAID CLI Reference]] for more details on how to use the CLI<br />
<br />
=== DELL (LSI-based) SAS controllers ===<br />
<br />
Here's what the output looks like when running verify.sh on a LSI-based card:<br />
<br />
<pre><br />
jail2 /mnt/data2# sh /root/verify.sh<br />
<br />
Adapter #0<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 0<br />
Device Id: 0<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018396142<br />
SAS Address(1): 0x0<br />
Connected Port Number: 0(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DESN<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 1<br />
Device Id: 1<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018395db2<br />
SAS Address(1): 0x0<br />
Connected Port Number: 1(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DERV<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 2<br />
Device Id: 2<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50006eece89<br />
SAS Address(1): 0x0<br />
Connected Port Number: 2(path0)<br />
Inquiry Data: SEAGATE ST3300555SS T2113LM4BFBZ<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 3<br />
Device Id: 3<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50006eee035<br />
SAS Address(1): 0x0<br />
Connected Port Number: 3(path0)<br />
Inquiry Data: SEAGATE ST3300555SS T2113LM4BGF7<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 4<br />
Device Id: 4<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50004bd7ea5<br />
SAS Address(1): 0x0<br />
Connected Port Number: 4(path0)<br />
Inquiry Data: SEAGATE ST3300656SS HS093QP0G8SW<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 5<br />
Device Id: 5<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e01f1c4112<br />
SAS Address(1): 0x0<br />
Connected Port Number: 5(path0)<br />
Inquiry Data: FUJITSU MBA3300RC D306BJ15P9201W06<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 0 (Target Id: 0)<br />
Name:<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:139392MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 1 (Target Id: 1)<br />
Name:MIRROR1<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 2 (Target Id: 2)<br />
Name:MIRROR2<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
Battery FRU : N/A<br />
Battery Warning : Enabled<br />
Memory Correctable Errors : 0<br />
Memory Uncorrectable Errors : 0<br />
BBU : Present<br />
BBU : Yes<br />
Cache When BBU Bad : Disabled<br />
press enter when ready to run verify<br />
<br />
</pre><br />
<br />
Before pressing enter, here's what we're looking for:<br />
<br />
<pre><br />
Enclosure Device ID: 32<br />
Slot Number: 0<br />
Device Id: 0<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018396142<br />
SAS Address(1): 0x0<br />
Connected Port Number: 0(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DESN<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
</pre><br />
<br />
This is the output shown for each physical drive in the system. We're looking to confirm it's Firmware state is ''Online'', and Media Error Count, Other Error Count, and Predictive Failure Count are all zero (or near zero).<br />
<br />
<pre><br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 1 (Target Id: 1)<br />
Name:MIRROR1<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
</pre><br />
<br />
This is the output for each logical drive. We're looking for State ''Optimal''. Also confirm <tt>Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU</tt><br />
<br />
<pre><br />
Exit Code: 0x00<br />
Battery FRU : N/A<br />
Battery Warning : Enabled<br />
Memory Correctable Errors : 0<br />
Memory Uncorrectable Errors : 0<br />
BBU : Present<br />
BBU : Yes<br />
Cache When BBU Bad : Disabled<br />
</pre><br />
<br />
Confirm that the battery is present and error-free.<br />
<br />
If all that checks out, you're ready to proceed with the verify. After pressing enter, the verify is started and here's what you see:<br />
<pre><br />
Start Check Consistency on Virtual Drive 0 (target id: 0) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Start Check Consistency on Virtual Drive 1 (target id: 1) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Start Check Consistency on Virtual Drive 2 (target id: 2) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Check Consistency<br />
<br />
Progress of Virtual Drives...<br />
<br />
Virtual Drive # Percent Complete Time Elps<br />
0 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:03<br />
1 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:02<br />
2 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:01<br />
<br />
Press <ESC> key to quit...<br />
</pre><br />
<br />
The progress for each drive is displayed until all drives have completed the verify. We just want to make sure that each drive goes to completion. No followup is needed...though there probably is a log or history where we can get more info.<br />
<br />
You will notice that jail7 does not run a verify- that's on purpose. The last time we tried this it crashed the system. So, this must be run from the BIOS (take the system offline for a couple hours).<br />
<br />
See [[RAIC_CLI#LSI|LSI RAID CLI Reference]] for more details on how to use the CLI<br />
<br />
=== LSI-based controllers (megaraid) ===<br />
<br />
There is a CLI for this however it's easier to do this with a curses GUI app: megaraid<br />
<br />
Currently only on these servers: virt15, virt16, and firewall2<br />
<br />
To run:<br />
<br />
# cd /usr/local/sbin/; megamgr<br />
<br />
Main menu:<br />
<pre>²ÚÄÄManagement MenuÄÄ¿²<br />
²³ Configure ³²<br />
²³ Initialize ³²<br />
²³ Objects ³²<br />
²³ Rebuild ³²<br />
²³ Check Consistency ³²<br />
²³ Advanced Menu ³²<br />
²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²</pre><br />
<br />
Before you check consistency, make sure the arrays are healthy.<br />
<br />
Objects -> Physical Drive<br />
<br />
Then look to make sure they're all <tt>ONLIN</tt><br />
<pre><br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄObjects - PHYSICAL DRIVE SELECTION MENUÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²<br />
²²²²²²²²³ ³²²²²²²<br />
²²²²²²²²³ Channel-1 ³²²²²²²<br />
²ÚÄÄMana³ ID ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»x ³²²²²²²<br />
²³ Confi³ 0º* ONLIN A01-01º ³²²²²²²<br />
²³ Initi³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²³ Objec³ 1º* ONLIN A01-02º ³²²²²²²<br />
²³ Rebui³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²³ Check³ 2º* ONLIN A02-01º ³²²²²²²<br />
²³ Advan³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²ÀÄÄÄÄÄij 3º* ONLIN A02-02º ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 4º* ONLIN A03-01º ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 5º* ONLIN A03-02ºþ ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 6º* º ³²²²²²²<br />
²²²²²²²²³ ÈÍÍÍÍÍÍÍÍÍÍÍÍÍͼx ³²²²²²²<br />
²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²<br />
²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²²²<br />
²²²²²²²²³Ch-1 ID-5 DISK 140013MB SEAGATE ST3146707LC 0003 ³²²²²²²²²<br />
²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²<br />
</pre><br />
<br />
<br />
Once that's done, hit escape once then the back arror to move back to the Objects menu.<br />
So you select Objects -> Logical Drive -> Logical Drive 1 -> Check Consistency -> YES<br />
<br />
<pre><br />
<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÚÄLogical Drives(02)Ä¿²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ Logical Drive 1 ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²ÚÄÄÄÄObjectsÄÄij Logical Drive 2 ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²ÚÄÄManagemen³ Adapter ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Configure ³ Logical Drive ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Initialize³ Physical Drive ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Objects ³ Channel ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Rebuild ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Check Consistency ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Advanced Menu ³²²²²²²²ÚÄÄÄÄLogical Drive 1ÄÄÄÄÄ¿²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²³ Initialize ÚÄCheck Consistency-1 ?Ä¿²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ Check Consiste³ YES ³²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ View/Update Pa³ NO ³²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³Select YES Or NO³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
</pre><br />
<br />
Then watch the progress. When done, escape back to Logical Drive then repeat for Logical Drive 2. If you ^C or accidentally escape out, you can come back in running the same commands and watch the progress again (it won't restart).<br />
<br />
You can exit megamgr by escaping out or ^C<br />
<br />
=== 3ware ===<br />
<br />
We are using 3ware controllers on backup1 & backup2. Running the verify script will give you different output based on the type of controller:<br />
<br />
<pre><br />
backup2 /d2# sh /root/verify.sh<br />
Controller: c0<br />
-------------<br />
Driver: 1.50.01.002<br />
Model: 7500-8<br />
FW: FE7X 1.05.00.068<br />
BIOS: BE7X 1.08.00.048<br />
Monitor: ME7X 1.01.00.040<br />
Serial #: F11605A3180172<br />
PCB: Rev3<br />
PCHIP: 1.30-33<br />
ACHIP: 3.20<br />
<br />
<br />
# of units: 3<br />
Unit 0: JBOD 186.31 GB ( 390721968 blocks): OK<br />
Unit 1: RAID 5 465.77 GB ( 976790016 blocks): DEGRADED<br />
Unit 5: RAID 5 698.65 GB ( 1465185024 blocks): DEGRADED<br />
<br />
# of ports: 8<br />
Port 0: WDC WD2000JB-00KFA0 WD-WCAMT1451690 186.31 GB (390721968 blocks): OK(unit 0)<br />
Port 1: WDC WD2500JB-00GVC0 WD-WCAL78219488 232.88 GB (488397168 blocks): OK(unit 1)<br />
Port 2: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT)<br />
Port 3: WDC WD2500JB-00GVC0 WD-WMAL73882417 232.88 GB (488397168 blocks): OK(unit 1)<br />
Port 4: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT)<br />
Port 5: WDC WD2500JB-00GVA0 WD-WMAL71338097 232.88 GB (488397168 blocks): OK(unit 5)<br />
Port 6: WDC WD2500JB-32EVA0 WD-WMAEH1301595 232.88 GB (488397168 blocks): OK(unit 5)<br />
Port 7: WDC WD2500JB-00GVC0 WD-WCAL78165566 232.88 GB (488397168 blocks): OK(unit 5)<br />
Controller: c1<br />
-------------<br />
Driver: 1.50.01.002<br />
Model: 7500-8<br />
FW: FE7X 1.05.00.068<br />
BIOS: BE7X 1.08.00.048<br />
Monitor: ME7X 1.01.00.040<br />
Serial #: F11605A3180167<br />
PCB: Rev3<br />
PCHIP: 1.30-33<br />
ACHIP: 3.20<br />
<br />
<br />
# of units: 2<br />
Unit 0: RAID 5 698.65 GB ( 1465185024 blocks): OK<br />
Unit 4: RAID 5 698.65 GB ( 1465185024 blocks): OK<br />
<br />
# of ports: 8<br />
Port 0: WDC WD2500JB-00GVA0 WD-WMAL71301258 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 1: WDC WD2500JB-00GVA0 WD-WMAL71322705 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 2: WDC WD2500JB-00GVA0 WD-WMAL71945050 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 3: WDC WD2500JB-00GVA0 WD-WMAL71316201 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 4: WDC WD2500JB-00GVC0 WD-WCAL78323749 232.88 GB (488397168 blocks): OK(unit 4)<br />
Port 5: WDC WD3200AAJB-00J3A0 WD-WCAV2V689068 298.09 GB (625142448 blocks): OK(unit 4)<br />
Port 6: WDC WD2500JB-00GVC0 WD-WCAL78234420 232.88 GB (488397168 blocks): OK(unit 4)<br />
Port 7: WDC WD2500JB-00GVC0 WD-WCAL78592213 232.88 GB (488397168 blocks): OK(unit 4)<br />
backup2 /d2#<br />
</pre><br />
<br />
On backup2 look for all ok, no verify.<br />
<br />
<pre><br />
[root@backup3 ~]# sh /root/verify.sh<br />
/c2 Driver Version = 1.26.02.002<br />
/c2 Model = 8006-2LP<br />
/c2 Available Memory = 512KB<br />
/c2 Firmware Version = FE8S 1.05.00.068<br />
/c2 Bios Version = BE7X 1.08.00.048<br />
/c2 Boot Loader Version = ME7X 1.01.00.040<br />
/c2 Serial Number = L018501C6481395<br />
/c2 PCB Version = Rev5<br />
/c2 PCHIP Version = 1.30-66<br />
/c2 ACHIP Version = 3.20<br />
/c2 Total Optimal Units = 1<br />
/c2 Not Optimal Units = 0<br />
<br />
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u1 RAID-1 OK - - - 931.512 ON -<br />
<br />
Port Status Unit Size Blocks Serial<br />
---------------------------------------------------------------<br />
p0 OK u1 931.51 GB 1953525168 WD-WMAW31148820<br />
p1 OK u1 931.51 GB 1953525168 WD-WCATR0277515<br />
<br />
<br />
Ctl Date Severity Alarm Message<br />
------------------------------------------------------------------------------<br />
<br />
Sending start verify message to /c2/u1 ... Done.<br />
<br />
<br />
when done run:<br />
tw_cli /c2 show alarms<br />
<br />
[root@backup3 ~]#<br />
</pre><br />
<br />
Automatically starts the verify, just run <tt>tw_cli /c2 show alarms</tt> as instructed to see the results of the verify.<br />
<br />
<pre><br />
[root@backup1 /data/deprecated]# sh /root/verify.sh<br />
/c0 Driver Version = 2.26.02.010<br />
/c0 Model = 9650SE-8LPML<br />
/c0 Available Memory = 224MB<br />
/c0 Firmware Version = FE9X 4.06.00.004<br />
/c0 Bios Version = BE9X 4.05.00.015<br />
/c0 Boot Loader Version = BL9X 3.08.00.001<br />
/c0 Serial Number = L326025A8270177<br />
/c0 PCB Version = Rev 032<br />
/c0 PCHIP Version = 2.00<br />
/c0 ACHIP Version = 1.90<br />
/c0 Number of Ports = 8<br />
/c0 Number of Drives = 6<br />
/c0 Number of Units = 1<br />
/c0 Total Optimal Units = 1<br />
/c0 Not Optimal Units = 0<br />
/c0 JBOD Export Policy = off<br />
/c0 Disk Spinup Policy = 1<br />
/c0 Spinup Stagger Time Policy (sec) = 1<br />
/c0 Auto-Carving Policy = off<br />
/c0 Auto-Carving Size = 2048 GB<br />
/c0 Auto-Rebuild Policy = on<br />
/c0 Controller Bus Type = PCIe<br />
/c0 Controller Bus Width = 1 lane<br />
/c0 Controller Bus Speed = 2.5 Gbps/lane<br />
<br />
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u0 RAID-5 OK - - 64K 4656.56 ON ON<br />
<br />
Port Status Unit Size Blocks Serial<br />
---------------------------------------------------------------<br />
p0 OK u0 931.51 GB 1953525168 9QJ1Y017<br />
p1 DEVICE-ERROR u0 931.51 GB 1953525168 9QJ1ZN07<br />
p2 OK u0 931.51 GB 1953525168 9QJ2XK1R<br />
p3 OK u0 931.51 GB 1953525168 9QJ2010B<br />
p4 OK u0 1.36 TB 2930277168 6XW0L36T<br />
p5 OK u0 931.51 GB 1953525168 WD-WMATV2444836<br />
p6 NOT-PRESENT - - - -<br />
p7 NOT-PRESENT - - - -<br />
<br />
<br />
Ctl Date Severity Alarm Message<br />
------------------------------------------------------------------------------<br />
c0 [Sat May 12 11:27:15 2012] WARNING Sector repair completed: port=0, LBA=0x6AE571C<br />
c0 [Sat May 12 19:16:21 2012] WARNING Sector repair completed: port=1, LBA=0x40E62A23<br />
c0 [Sat May 12 21:40:56 2012] INFO Verify completed: unit=0<br />
c0 [Mon May 14 00:53:53 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFA7<br />
c0 [Mon May 14 00:58:21 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAA<br />
c0 [Mon May 14 04:35:13 2012] WARNING Sector repair completed: port=0, LBA=0x8FEF2CF<br />
c0 [Mon May 14 04:38:22 2012] WARNING Sector repair completed: port=0, LBA=0x8FEF2D1<br />
c0 [Tue May 15 22:53:46 2012] WARNING Sector repair completed: port=0, LBA=0x13C2622<br />
c0 [Wed May 16 00:39:31 2012] WARNING Sector repair completed: port=0, LBA=0x365A67F<br />
c0 [Wed May 16 00:39:37 2012] WARNING Sector repair completed: port=0, LBA=0x365A685<br />
c0 [Wed May 16 00:47:18 2012] WARNING Sector repair completed: port=0, LBA=0x365A687<br />
c0 [Sat May 19 00:01:44 2012] INFO Verify started: unit=0<br />
c0 [Sat May 19 04:46:20 2012] WARNING Sector repair completed: port=0, LBA=0x365A68E<br />
c0 [Sat May 19 13:37:06 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAC<br />
c0 [Sat May 19 13:37:28 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAE<br />
c0 [Sat May 19 13:37:47 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFB1<br />
c0 [Sat May 19 13:38:00 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFB3<br />
c0 [Sat May 19 21:47:45 2012] INFO Verify completed: unit=0<br />
c0 [Wed May 23 12:21:41 2012] INFO Cache synchronization completed: unit=0<br />
c0 [Fri May 25 00:08:19 2012] WARNING Sector repair completed: port=0, LBA=0x12DA76C<br />
c0 [Fri May 25 00:08:34 2012] WARNING Sector repair completed: port=0, LBA=0x12E4901<br />
c0 [Fri May 25 00:09:33 2012] WARNING Sector repair completed: port=0, LBA=0x12DA773<br />
c0 [Fri May 25 00:39:12 2012] WARNING Sector repair completed: port=0, LBA=0x42C597B<br />
c0 [Sat May 26 00:01:45 2012] INFO Verify started: unit=0<br />
c0 [Sat May 26 00:42:05 2012] WARNING Sector repair completed: port=1, LBA=0x323C1AC<br />
c0 [Sat May 26 00:51:43 2012] WARNING Sector repair completed: port=1, LBA=0x323C1AE<br />
c0 [Sat May 26 01:54:21 2012] WARNING Sector repair completed: port=1, LBA=0x2F0D302<br />
c0 [Sat May 26 02:06:38 2012] WARNING Sector repair completed: port=0, LBA=0x12DA777<br />
c0 [Sat May 26 02:07:21 2012] WARNING Sector repair completed: port=0, LBA=0x12E48FE<br />
c0 [Sat May 26 04:20:00 2012] WARNING Sector repair completed: port=1, LBA=0x2F0D306<br />
c0 [Sat May 26 04:32:58 2012] WARNING Sector repair completed: port=1, LBA=0x323C1B1<br />
c0 [Sat May 26 04:33:21 2012] WARNING Sector repair completed: port=1, LBA=0x323C1B3<br />
c0 [Sat May 26 04:33:44 2012] WARNING Sector repair completed: port=1, LBA=0x323C1BA<br />
c0 [Sat May 26 05:24:07 2012] WARNING Sector repair completed: port=1, LBA=0x3F83862<br />
c0 [Sat May 26 05:25:09 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 06:08:13 2012] WARNING Sector repair completed: port=0, LBA=0x4CDC6A2<br />
c0 [Sat May 26 09:49:35 2012] WARNING Sector repair completed: port=1, LBA=0x6CACD4A<br />
c0 [Sat May 26 18:10:44 2012] WARNING Sector repair completed: port=1, LBA=0x18F425EA<br />
c0 [Sat May 26 19:45:40 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:22:52 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:15 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:22 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:35 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:41 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:49 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:57 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:24:02 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:54:41 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 22:00:30 2012] INFO Verify completed: unit=0<br />
c0 [Sat Jun 2 00:01:43 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 2 00:30:17 2012] WARNING Sector repair completed: port=0, LBA=0x2B911E4<br />
c0 [Sat Jun 2 00:50:57 2012] WARNING Sector repair completed: port=0, LBA=0x5A807CA6<br />
c0 [Sat Jun 2 04:13:13 2012] WARNING Sector repair completed: port=0, LBA=0x2D18291<br />
c0 [Sat Jun 2 04:13:35 2012] WARNING Sector repair completed: port=0, LBA=0x2D1829F<br />
c0 [Sat Jun 2 21:48:02 2012] INFO Verify completed: unit=0<br />
c0 [Mon Jun 4 04:40:34 2012] WARNING Sector repair completed: port=1, LBA=0x4AF8098F<br />
c0 [Tue Jun 5 00:28:19 2012] WARNING Sector repair completed: port=1, LBA=0x263C5CD<br />
c0 [Tue Jun 5 00:33:06 2012] WARNING Sector repair completed: port=1, LBA=0x263C5CF<br />
c0 [Thu Jun 7 00:34:27 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B5F<br />
c0 [Thu Jun 7 00:38:50 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B61<br />
c0 [Fri Jun 8 00:07:13 2012] WARNING Sector repair completed: port=0, LBA=0xC131F6B<br />
c0 [Sat Jun 9 00:01:41 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 9 00:29:11 2012] WARNING Sector repair completed: port=0, LBA=0x6C7614D<br />
c0 [Sat Jun 9 00:38:25 2012] WARNING Sector repair completed: port=0, LBA=0x6C76152<br />
c0 [Sat Jun 9 04:02:30 2012] WARNING Sector repair completed: port=1, LBA=0x263C5D1<br />
c0 [Sat Jun 9 04:02:52 2012] WARNING Sector repair completed: port=1, LBA=0x263C5D3<br />
c0 [Sat Jun 9 04:07:32 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E12<br />
c0 [Sat Jun 9 04:07:57 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E15<br />
c0 [Sat Jun 9 04:08:16 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E17<br />
c0 [Sat Jun 9 04:08:45 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E19<br />
c0 [Sat Jun 9 04:15:04 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B64<br />
c0 [Sat Jun 9 04:15:26 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B66<br />
c0 [Sat Jun 9 04:15:45 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B68<br />
c0 [Sat Jun 9 04:15:59 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B6C<br />
c0 [Sat Jun 9 04:16:13 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B6E<br />
c0 [Sat Jun 9 21:48:52 2012] INFO Verify completed: unit=0<br />
c0 [Thu Jun 14 00:40:10 2012] WARNING Sector repair completed: port=0, LBA=0x334F14B<br />
c0 [Sat Jun 16 00:01:38 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 16 21:16:19 2012] INFO Verify completed: unit=0<br />
c0 [Tue Jun 19 02:03:43 2012] WARNING Sector repair completed: port=1, LBA=0xFE41EAD<br />
c0 [Wed Jun 20 02:30:02 2012] WARNING Sector repair completed: port=1, LBA=0xD99145C<br />
c0 [Sat Jun 23 00:01:36 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 23 04:27:04 2012] WARNING Sector repair completed: port=1, LBA=0x2FAD311<br />
c0 [Sat Jun 23 06:52:38 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC8D<br />
c0 [Sat Jun 23 06:53:03 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC91<br />
c0 [Sat Jun 23 06:53:21 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC94<br />
c0 [Sat Jun 23 17:00:22 2012] WARNING Sector repair completed: port=1, LBA=0xF9AC7C9<br />
c0 [Sat Jun 23 21:15:19 2012] INFO Verify completed: unit=0<br />
c0 [Sat Jun 30 00:01:34 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 30 05:24:13 2012] WARNING Sector repair completed: port=0, LBA=0x3FAA9E7<br />
c0 [Sat Jun 30 14:49:39 2012] WARNING Sector repair completed: port=1, LBA=0x869931C<br />
c0 [Sat Jun 30 21:31:05 2012] INFO Verify completed: unit=0<br />
c0 [Tue Jul 3 03:40:25 2012] WARNING Sector repair completed: port=1, LBA=0xD36C7F7<br />
c0 [Fri Jul 6 02:50:18 2012] WARNING Sector repair completed: port=1, LBA=0x3562470<br />
c0 [Fri Jul 6 22:18:26 2012] WARNING Sector repair completed: port=1, LBA=0x3563173<br />
c0 [Sat Jul 7 00:01:31 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 7 00:50:16 2012] WARNING Sector repair completed: port=0, LBA=0x76EE88<br />
c0 [Sat Jul 7 00:50:39 2012] WARNING Sector repair completed: port=0, LBA=0x76EE8F<br />
c0 [Sat Jul 7 21:39:36 2012] INFO Verify completed: unit=0<br />
c0 [Sun Jul 8 02:51:05 2012] WARNING Sector repair completed: port=0, LBA=0x67759D<br />
c0 [Sun Jul 8 02:53:55 2012] WARNING Sector repair completed: port=0, LBA=0x67759B<br />
c0 [Tue Jul 10 16:17:21 2012] WARNING Sector repair completed: port=0, LBA=0x15C8C695<br />
c0 [Wed Jul 11 22:51:22 2012] WARNING Sector repair completed: port=1, LBA=0x355BBD0<br />
c0 [Sat Jul 14 00:01:28 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 14 01:33:40 2012] WARNING Sector repair completed: port=1, LBA=0x1333BCF4<br />
c0 [Sat Jul 14 03:36:23 2012] WARNING Sector repair completed: port=1, LBA=0x2174773<br />
c0 [Sat Jul 14 11:26:44 2012] WARNING Sector repair completed: port=0, LBA=0x7429AB7<br />
c0 [Sat Jul 14 16:53:50 2012] WARNING Sector repair completed: port=1, LBA=0xA17EB3F<br />
c0 [Sat Jul 14 21:19:25 2012] INFO Verify completed: unit=0<br />
c0 [Wed Jul 18 05:08:47 2012] WARNING Sector repair completed: port=1, LBA=0x17D62EDC<br />
c0 [Wed Jul 18 05:14:15 2012] WARNING Sector repair completed: port=1, LBA=0x17D62EE1<br />
c0 [Thu Jul 19 03:24:59 2012] WARNING Sector repair completed: port=0, LBA=0x7733C3D<br />
c0 [Thu Jul 19 03:25:20 2012] WARNING Sector repair completed: port=0, LBA=0x773CEA5<br />
c0 [Thu Jul 19 03:28:16 2012] WARNING Sector repair completed: port=0, LBA=0x7733C42<br />
c0 [Thu Jul 19 03:28:41 2012] WARNING Sector repair completed: port=0, LBA=0x773CEAF<br />
c0 [Sat Jul 21 00:01:26 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 21 03:07:30 2012] WARNING Sector repair completed: port=1, LBA=0x1CC6936<br />
c0 [Sat Jul 21 03:07:52 2012] WARNING Sector repair completed: port=1, LBA=0x1CC6938<br />
c0 [Sat Jul 21 03:08:11 2012] WARNING Sector repair completed: port=1, LBA=0x1CC693A<br />
c0 [Sat Jul 21 16:43:56 2012] WARNING Sector repair completed: port=0, LBA=0xD04C914<br />
c0 [Sat Jul 21 16:45:31 2012] WARNING Sector repair completed: port=1, LBA=0xD456973<br />
c0 [Sat Jul 21 21:14:29 2012] INFO Verify completed: unit=0<br />
c0 [Wed Jul 25 03:37:25 2012] WARNING Sector repair completed: port=0, LBA=0x1F8E6C43<br />
c0 [Sat Jul 28 00:01:24 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 28 01:45:27 2012] WARNING Sector repair completed: port=0, LBA=0x11584AD<br />
c0 [Sat Jul 28 18:54:25 2012] WARNING Sector repair completed: port=1, LBA=0x447C3E6C<br />
c0 [Sat Jul 28 21:13:46 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 1 03:20:11 2012] WARNING Sector repair completed: port=0, LBA=0x805FEF<br />
c0 [Fri Aug 3 00:50:03 2012] WARNING Sector repair completed: port=0, LBA=0xCED0ACA<br />
c0 [Sat Aug 4 00:01:22 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 4 00:52:51 2012] WARNING Sector repair completed: port=0, LBA=0x805FF3<br />
c0 [Sat Aug 4 00:53:14 2012] WARNING Sector repair completed: port=0, LBA=0x805FF5<br />
c0 [Sat Aug 4 00:53:33 2012] WARNING Sector repair completed: port=0, LBA=0x805FF7<br />
c0 [Sat Aug 4 00:53:47 2012] WARNING Sector repair completed: port=0, LBA=0x805FF9<br />
c0 [Sat Aug 4 00:54:00 2012] WARNING Sector repair completed: port=0, LBA=0x805FFB<br />
c0 [Sat Aug 4 00:54:14 2012] WARNING Sector repair completed: port=0, LBA=0x805FFD<br />
c0 [Sat Aug 4 00:54:27 2012] WARNING Sector repair completed: port=0, LBA=0x805FFF<br />
c0 [Sat Aug 4 04:43:12 2012] WARNING Sector repair completed: port=1, LBA=0x16974289<br />
c0 [Sat Aug 4 04:58:17 2012] WARNING Sector repair completed: port=1, LBA=0x1697428E<br />
c0 [Sat Aug 4 20:54:53 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 8 03:21:55 2012] ERROR Drive timeout detected: port=1<br />
c0 [Wed Aug 8 15:31:44 2012] WARNING Sector repair completed: port=0, LBA=0x1A366CD3<br />
c0 [Sat Aug 11 00:01:21 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 11 20:40:51 2012] INFO Verify completed: unit=0<br />
c0 [Thu Aug 16 05:10:55 2012] WARNING Sector repair completed: port=0, LBA=0x1C22593<br />
c0 [Sat Aug 18 00:01:18 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 18 03:00:20 2012] WARNING Sector repair completed: port=0, LBA=0x1C225A5<br />
c0 [Sat Aug 18 03:43:00 2012] WARNING Sector repair completed: port=1, LBA=0x23EE91E<br />
c0 [Sat Aug 18 03:43:23 2012] WARNING Sector repair completed: port=1, LBA=0x23EE920<br />
c0 [Sat Aug 18 17:00:06 2012] WARNING Sector repair completed: port=1, LBA=0x137D066A<br />
c0 [Sat Aug 18 17:00:29 2012] WARNING Sector repair completed: port=1, LBA=0x137D066D<br />
c0 [Sat Aug 18 21:13:01 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 22 01:36:08 2012] WARNING Sector repair completed: port=0, LBA=0x2560A0F<br />
c0 [Wed Aug 22 01:37:42 2012] WARNING Sector repair completed: port=0, LBA=0x2560A13<br />
c0 [Fri Aug 24 04:01:36 2012] WARNING Sector repair completed: port=1, LBA=0x55C1A5DF<br />
c0 [Fri Aug 24 05:02:06 2012] WARNING Sector repair completed: port=1, LBA=0xCE3378A<br />
c0 [Sat Aug 25 00:01:17 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 25 00:31:06 2012] WARNING Sector repair completed: port=1, LBA=0x50F65D<br />
c0 [Sat Aug 25 00:39:52 2012] WARNING Sector repair completed: port=0, LBA=0x678FF4<br />
c0 [Sat Aug 25 03:43:15 2012] WARNING Sector repair completed: port=0, LBA=0x2560A15<br />
c0 [Sat Aug 25 03:43:39 2012] WARNING Sector repair completed: port=0, LBA=0x2560A19<br />
c0 [Sat Aug 25 03:43:58 2012] WARNING Sector repair completed: port=0, LBA=0x2560A1B<br />
c0 [Sat Aug 25 03:44:30 2012] WARNING Sector repair completed: port=0, LBA=0x2560A21<br />
c0 [Sat Aug 25 20:58:14 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 29 04:57:15 2012] WARNING Sector repair completed: port=1, LBA=0xF3957EB<br />
c0 [Sat Sep 1 00:01:15 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 1 03:21:52 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC86<br />
c0 [Sat Sep 1 03:22:15 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC88<br />
c0 [Sat Sep 1 03:22:34 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC8A<br />
c0 [Sat Sep 1 03:22:47 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC8C<br />
c0 [Sat Sep 1 17:17:22 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD1<br />
c0 [Sat Sep 1 17:17:45 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD3<br />
c0 [Sat Sep 1 17:18:04 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD5<br />
c0 [Sat Sep 1 21:36:56 2012] INFO Verify completed: unit=0<br />
c0 [Thu Sep 6 00:07:30 2012] WARNING Sector repair completed: port=0, LBA=0xDA3C64B<br />
c0 [Thu Sep 6 00:32:56 2012] WARNING Sector repair completed: port=1, LBA=0x6BBA816<br />
c0 [Sat Sep 8 00:01:13 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 8 00:09:56 2012] WARNING Sector repair completed: port=0, LBA=0xDEBC958<br />
c0 [Sat Sep 8 04:38:45 2012] WARNING Sector repair completed: port=0, LBA=0x38D254F<br />
c0 [Sat Sep 8 20:44:50 2012] INFO Verify completed: unit=0<br />
c0 [Mon Sep 10 01:26:34 2012] WARNING Sector repair completed: port=1, LBA=0xFFD8D5E<br />
c0 [Wed Sep 12 00:33:48 2012] WARNING Sector repair completed: port=1, LBA=0xE8DB928<br />
c0 [Wed Sep 12 00:36:33 2012] WARNING Sector repair completed: port=1, LBA=0x6D49411<br />
c0 [Fri Sep 14 01:59:39 2012] WARNING Sector repair completed: port=0, LBA=0x1467F1C<br />
c0 [Fri Sep 14 02:08:27 2012] WARNING Sector repair completed: port=1, LBA=0x14C8ABD<br />
c0 [Fri Sep 14 03:54:47 2012] WARNING Sector repair completed: port=0, LBA=0x1580C915<br />
c0 [Sat Sep 15 00:01:11 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 15 02:38:14 2012] WARNING Sector repair completed: port=0, LBA=0x1C178973<br />
c0 [Sat Sep 15 02:59:02 2012] WARNING Sector repair completed: port=0, LBA=0x1C178975<br />
c0 [Sat Sep 15 04:47:08 2012] WARNING Sector repair completed: port=0, LBA=0x3FA0356<br />
c0 [Sat Sep 15 04:47:31 2012] WARNING Sector repair completed: port=0, LBA=0x3FA0359<br />
c0 [Sat Sep 15 10:41:59 2012] WARNING Sector repair completed: port=0, LBA=0x6DFD1EC<br />
c0 [Sat Sep 15 13:25:23 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD100<br />
c0 [Sat Sep 15 13:25:31 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD104<br />
c0 [Sat Sep 15 13:25:54 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD106<br />
c0 [Sat Sep 15 17:10:50 2012] WARNING Sector repair completed: port=0, LBA=0x1C178977<br />
c0 [Sat Sep 15 20:59:57 2012] INFO Verify completed: unit=0<br />
c0 [Tue Sep 18 01:17:18 2012] WARNING Sector repair completed: port=1, LBA=0x803B05B<br />
c0 [Sat Sep 22 00:01:10 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 22 20:54:31 2012] INFO Verify completed: unit=0<br />
c0 [Tue Sep 25 01:56:47 2012] WARNING Sector repair completed: port=0, LBA=0x26E3909<br />
c0 [Sat Sep 29 00:01:08 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 29 02:04:14 2012] WARNING Sector repair completed: port=0, LBA=0x146AC03<br />
c0 [Sat Sep 29 10:58:39 2012] WARNING Sector repair completed: port=0, LBA=0x6D4EB0E<br />
c0 [Sat Sep 29 10:59:02 2012] WARNING Sector repair completed: port=0, LBA=0x6D4EB14<br />
c0 [Sat Sep 29 11:22:44 2012] WARNING Sector repair completed: port=0, LBA=0x6F79623<br />
c0 [Sat Sep 29 13:50:48 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D65E<br />
c0 [Sat Sep 29 13:51:11 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D661<br />
c0 [Sat Sep 29 13:51:30 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D663<br />
c0 [Sat Sep 29 20:57:34 2012] INFO Verify completed: unit=0<br />
c0 [Mon Oct 1 04:47:24 2012] WARNING Sector repair completed: port=0, LBA=0xC5BC6F2<br />
c0 [Tue Oct 2 02:00:27 2012] WARNING Sector repair completed: port=0, LBA=0x1547667<br />
c0 [Tue Oct 2 02:01:56 2012] WARNING Sector repair completed: port=0, LBA=0x154766F<br />
c0 [Tue Oct 2 05:02:31 2012] WARNING Sector repair completed: port=1, LBA=0xD67D054<br />
c0 [Tue Oct 2 05:04:14 2012] WARNING Sector repair completed: port=1, LBA=0xD67D056<br />
c0 [Wed Oct 3 01:22:12 2012] WARNING Sector repair completed: port=1, LBA=0x12AAF8CA<br />
c0 [Thu Oct 4 04:29:22 2012] WARNING Sector repair completed: port=0, LBA=0x13E6F992<br />
c0 [Thu Oct 4 05:10:51 2012] WARNING Sector repair completed: port=0, LBA=0x1C252A4<br />
c0 [Sat Oct 6 00:01:07 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 6 19:41:18 2012] WARNING Sector repair completed: port=1, LBA=0x5A5C3AE8<br />
c0 [Sat Oct 6 21:01:05 2012] INFO Verify completed: unit=0<br />
c0 [Mon Oct 8 00:32:06 2012] WARNING Sector repair completed: port=0, LBA=0x6C60D3E<br />
c0 [Tue Oct 9 03:51:03 2012] WARNING Sector repair completed: port=1, LBA=0x89B5EC9<br />
c0 [Thu Oct 11 04:21:17 2012] WARNING Sector repair completed: port=1, LBA=0x13F85833<br />
c0 [Sat Oct 13 00:01:05 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 13 05:12:40 2012] WARNING Sector repair completed: port=0, LBA=0x3FA5134<br />
c0 [Sat Oct 13 21:08:35 2012] INFO Verify completed: unit=0<br />
c0 [Tue Oct 16 03:53:50 2012] WARNING Sector repair completed: port=1, LBA=0x148AA1BD<br />
c0 [Thu Oct 18 03:20:30 2012] WARNING Sector repair completed: port=0, LBA=0x1C8DABCB<br />
c0 [Thu Oct 18 04:52:50 2012] WARNING Sector repair completed: port=0, LBA=0xE879057<br />
c0 [Sat Oct 20 00:01:04 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 20 02:19:25 2012] WARNING Sector repair completed: port=1, LBA=0x174B012<br />
c0 [Sat Oct 20 03:41:38 2012] WARNING Sector repair completed: port=0, LBA=0x256D93B<br />
c0 [Sat Oct 20 03:42:01 2012] WARNING Sector repair completed: port=0, LBA=0x256D93D<br />
c0 [Sat Oct 20 03:42:40 2012] WARNING Sector repair completed: port=0, LBA=0x256D940<br />
c0 [Sat Oct 20 03:42:59 2012] WARNING Sector repair completed: port=0, LBA=0x256D942<br />
c0 [Sat Oct 20 03:43:12 2012] WARNING Sector repair completed: port=0, LBA=0x256D944<br />
c0 [Sat Oct 20 03:43:26 2012] WARNING Sector repair completed: port=0, LBA=0x256D948<br />
c0 [Sat Oct 20 16:37:52 2012] WARNING Sector repair completed: port=0, LBA=0xE879060<br />
c0 [Sat Oct 20 16:38:15 2012] WARNING Sector repair completed: port=0, LBA=0xE879062<br />
c0 [Sat Oct 20 21:00:18 2012] INFO Verify completed: unit=0<br />
c0 [Sat Oct 20 23:49:01 2012] WARNING Sector repair completed: port=1, LBA=0x4473E908<br />
c0 [Sun Oct 21 03:42:26 2012] WARNING Sector repair completed: port=1, LBA=0x175BADD5<br />
c0 [Tue Oct 23 01:09:04 2012] WARNING Sector repair completed: port=1, LBA=0x6E524860<br />
c0 [Fri Oct 26 03:21:25 2012] WARNING Sector repair completed: port=0, LBA=0x802C61<br />
c0 [Fri Oct 26 04:22:21 2012] WARNING Sector repair completed: port=0, LBA=0x176353CD<br />
c0 [Sat Oct 27 00:01:03 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 27 00:49:35 2012] WARNING Sector repair completed: port=0, LBA=0x802C65<br />
c0 [Sat Oct 27 17:02:24 2012] WARNING Sector repair completed: port=1, LBA=0xC1FF26D<br />
c0 [Sat Oct 27 17:09:06 2012] WARNING Sector repair completed: port=0, LBA=0xDF621AD<br />
c0 [Sat Oct 27 21:30:57 2012] INFO Verify completed: unit=0<br />
c0 [Tue Oct 30 00:20:46 2012] WARNING Sector repair completed: port=0, LBA=0xE9FE2AB<br />
c0 [Wed Oct 31 02:02:03 2012] WARNING Sector repair completed: port=0, LBA=0x1460C25<br />
c0 [Wed Oct 31 02:04:05 2012] WARNING Sector repair completed: port=0, LBA=0x1460C28<br />
c0 [Thu Nov 1 00:48:34 2012] WARNING Sector repair completed: port=1, LBA=0xA7C92BE<br />
c0 [Thu Nov 1 05:04:45 2012] WARNING Sector repair completed: port=0, LBA=0x1C252C2<br />
<br />
[root@backup1 /data/deprecated]#<br />
</pre><br />
<br />
Look for failed drives and errors. Obviously from the above we need to probably replace drives 0 and 1 and drive 1 is even showing as having problems, yet the RAID array is healthy, amazingly. You also see the automatic verifies.<br />
<br />
Note: when rebuilding a degraded mirror, you will see no progress as it rebuilds in the cli<br />
<pre>Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u0 RAID-5 REBUILDING 0 - 64K 4656.56 OFF ON</pre><br />
<br />
<br />
=== areca ===<br />
<br />
We are using an areca controller on backup3. <br />
<br />
<pre><br />
[root@newbackup3 ~]# sh /root/verify.sh<br />
# Name Raid Name Level Capacity Ch/Id/Lun State<br />
===============================================================================<br />
1 ARC-1160-VOL#00 Raid Set # 00 Raid5 5000.0GB 00/00/00 Checking(19.7%)<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
# Name Disks TotalCap FreeCap DiskChannels State<br />
===============================================================================<br />
1 Raid Set # 00 6 6000.0GB 0.0GB 123456 Checking<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
Date-Time Device Event Type Elapsed Time Errors<br />
===============================================================================<br />
2012-12-05 20:40:58 ARC-1160-VOL#00 Start Checking<br />
2012-12-01 05:06:04 ARC-1160-VOL#00 Complete Init 027:30:45<br />
2012-11-30 01:35:19 ARC-1160-VOL#00 Start Initialize<br />
2026-08-06 01:34:52 H/W MONITOR Raid Powered On<br />
2012-11-30 01:33:36 ARC-1160-VOL#00 Stop Initialization 000:31:48<br />
2012-11-30 01:01:47 ARC-1160-VOL#00 Start Initialize<br />
2026-08-06 00:58:13 H/W MONITOR Raid Powered On<br />
2012-11-30 00:57:26 ARC-1160-VOL#00 Stop Initialization 000:57:07<br />
2012-11-30 00:00:19 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 23:56:48 H/W MONITOR Raid Powered On<br />
2026-08-05 23:52:58 H/W MONITOR Raid Powered On<br />
2026-08-05 23:50:14 H/W MONITOR Raid Powered On<br />
2026-08-05 23:43:30 H/W MONITOR Raid Powered On<br />
2012-11-29 23:10:07 ARC-1160-VOL#00 Stop Initialization 000:00:56<br />
2012-11-29 23:09:11 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 23:08:57 H/W MONITOR Raid Powered On<br />
2012-11-29 23:08:10 ARC-1160-VOL#00 Stop Initialization 000:20:41<br />
2012-11-29 22:47:29 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 22:46:59 H/W MONITOR Raid Powered On<br />
2026-08-05 22:45:55 H/W MONITOR Raid Powered On<br />
2026-08-05 22:44:53 H/W MONITOR Raid Powered On<br />
2026-08-05 22:42:06 H/W MONITOR Raid Powered On<br />
2026-08-05 22:40:50 H/W MONITOR Raid Powered On<br />
2012-11-29 22:40:04 ARC-1160-VOL#00 Stop Initialization 000:24:25<br />
2012-11-29 22:15:38 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 22:15:11 000:000001215B00 Restart Init LBA Point<br />
2026-08-05 22:15:10 H/W MONITOR Raid Powered On<br />
2012-11-29 21:56:38 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 21:56:12 H/W MONITOR Raid Powered On<br />
2026-08-05 21:56:04 IDE Channel #03 Device Inserted<br />
2012-11-29 21:55:13 IDE Channel #04 Device Inserted<br />
2012-11-29 21:55:03 IDE Channel #02 Device Inserted<br />
2026-08-05 21:53:09 H/W MONITOR Raid Powered On<br />
2026-08-05 20:51:46 H/W MONITOR Raid Powered On<br />
2026-08-05 20:49:56 H/W MONITOR Raid Powered On<br />
2026-08-05 20:48:29 H/W MONITOR Raid Powered On<br />
2026-08-05 20:46:29 H/W MONITOR Raid Powered On<br />
2026-08-05 20:44:49 H/W MONITOR Raid Powered On<br />
2026-08-05 20:43:01 H/W MONITOR Raid Powered On<br />
2026-08-05 20:36:25 H/W MONITOR Raid Powered On<br />
2026-08-05 20:31:18 H/W MONITOR Raid Powered On<br />
2026-08-05 20:30:08 H/W MONITOR Raid Powered On<br />
2026-08-05 20:08:40 H/W MONITOR Raid Powered On<br />
2026-08-05 20:06:11 H/W MONITOR Raid Powered On<br />
2026-08-05 20:05:14 H/W MONITOR Raid Powered On<br />
2026-08-05 20:03:58 H/W MONITOR Raid Powered On<br />
2026-08-05 20:00:56 H/W MONITOR Raid Powered On<br />
2026-08-05 19:57:57 H/W MONITOR Raid Powered On<br />
2026-08-05 19:56:15 H/W MONITOR Raid Powered On<br />
2026-08-05 19:55:05 H/W MONITOR Raid Powered On<br />
2026-08-05 17:24:36 H/W MONITOR Raid Powered On<br />
2026-08-05 17:22:43 H/W MONITOR Raid Powered On<br />
2026-08-05 04:50:42 H/W MONITOR Raid Powered On<br />
2026-08-05 04:47:33 H/W MONITOR Raid Powered On<br />
2026-08-05 04:43:57 H/W MONITOR Raid Powered On<br />
2026-08-05 04:18:52 H/W MONITOR Raid Powered On<br />
2026-08-05 04:17:30 H/W MONITOR Raid Powered On<br />
2026-08-05 04:13:30 H/W MONITOR Raid Powered On<br />
2026-08-05 04:10:26 H/W MONITOR Raid Powered On<br />
2026-08-05 04:09:23 H/W MONITOR Raid Powered On<br />
2026-08-05 00:08:09 H/W MONITOR Raid Powered On<br />
2026-08-05 00:07:12 H/W MONITOR Raid Powered On<br />
2026-08-05 00:05:51 H/W MONITOR Raid Powered On<br />
2026-08-05 00:04:27 H/W MONITOR Raid Powered On<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
press enter when ready to run verify<br />
</pre><br />
<br />
Look for failed drives and errors. <br />
<br />
When it proceed's to verifying, you can confirm with:<br />
<br />
<pre><br />
[root@newbackup3 ~]# cli64 vsf info<br />
# Name Raid Name Level Capacity Ch/Id/Lun State<br />
===============================================================================<br />
1 ARC-1160-VOL#00 Raid Set # 00 Raid5 5000.0GB 00/00/00 Checking(22.5%)<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
[root@newbackup3 ~]#<br />
</pre><br />
<br />
== Update OS list ==<br />
<br />
# check for any new VZ templates we want to offer: <tt>vzup2date -z</tt><br />
# see if there's any OS's we want to include in our colo install list. Update 2 places: <tt>signup/html/colo_quote.html</tt> & <tt>signup/html/step1.html</tt><br />
# update the mgmt database (ref_templates table, ref_systems table).<br />
<br />
= Infrequent tasks =<br />
<br />
== Free up space on gateway ==<br />
<br />
<pre>newgateway /var/spool# cd clientmqueue/<br />
newgateway /var/spool/clientmqueue# sh<br />
# for f in `ls`; do rm $f; done<br />
exit</pre><br />
<br />
== Free up space on mail ==<br />
<br />
You can clear out root mail:<br />
<pre><br />
mail /var/log# ll -h /var/mail/root<br />
-rw------- 1 root mail 543K Dec 19 13:05 /var/mail/root<br />
mail /var/log# rm /var/mail/root<br />
</pre><br />
<br />
Or you can archive mail logs:<br />
<pre><br />
mail /var/log# ls -l htt*<br />
-rw-r--r-- 1 root wheel 297436931 Dec 19 13:26 httpd-access.log<br />
-rw-r--r-- 1 root wheel 9824324 Jul 4 11:34 httpd-access.log.old.0.gz<br />
-rw-r--r-- 1 root wheel 6884137 Mar 17 2012 httpd-access.log.old.1.gz<br />
-rw-r--r-- 1 root wheel 18557444 Dec 3 2009 httpd-access.log.old.10.gz<br />
-rw-r--r-- 1 root wheel 14740263 Jan 9 2007 httpd-access.log.old.11.gz<br />
-rw-r--r-- 1 root wheel 14209465 Nov 28 2007 httpd-access.log.old.12.gz<br />
-rw-r--r-- 1 root wheel 16874396 Feb 19 2012 httpd-access.log.old.3.gz<br />
-rw-r--r-- 1 root wheel 14554859 Jul 22 2011 httpd-access.log.old.4.gz<br />
-rw-r--r-- 1 root wheel 10513227 Feb 18 2011 httpd-access.log.old.5.gz<br />
-rw-r--r-- 1 root wheel 7201946 Oct 29 2010 httpd-access.log.old.6.gz<br />
-rw-r--r-- 1 root wheel 10062537 May 6 2010 httpd-access.log.old.7.gz<br />
-rw-r--r-- 1 root wheel 10157042 Aug 12 2010 httpd-access.log.old.8.gz<br />
-rw-r--r-- 1 root wheel 11909534 Mar 4 2010 httpd-access.log.old.9.gz<br />
-rw-r--r-- 1 root wheel 59030930 Dec 19 13:01 httpd-error.log<br />
-rw-r--r-- 1 root wheel 3413134 Mar 4 2010 httpd-error.log.0.gz<br />
-rw-r--r-- 1 root wheel 795515 May 1 2007 httpd-error.log.1.gz<br />
-rw-r--r-- 1 root wheel 1142153 Nov 30 2007 httpd-error.log.2.gz<br />
-rw-r--r-- 1 root wheel 2325801 Feb 18 2011 httpd-error.log.gz<br />
<br />
mail /var/log# sh<br />
# for f in 12 11 10 9 8 7 6 5 4 3 2 1 0; do g=`echo $f+1|bc`; mv httpd-access.log.old.$f.gz httpd-access.log.old.$g.gz; done<br />
# mv httpd-access.log httpd-access.log.old.0<br />
# touch httpd-access.log<br />
# apachectl restart<br />
# gzip httpd-access.log.old.0<br />
<br />
# for f in 2 1 0; do g=`echo $f+1|bc`; mv httpd-error.log.$f.gz httpd-error.log.$g.gz; done<br />
# mv httpd-error.log httpd-error.log.0<br />
# touch httpd-error.log<br />
# apachectl restart<br />
# gzip httpd-error.log.0<br />
# exit<br />
</pre><br />
<br />
== Free up space on bwdb2 ==<br />
<br />
You can either remove items from <tt>/usr/home/archive</tt> or you can scp them to backup3:/data/bwdb2/archive .<br />
<br />
== Free up space on backup1 ==<br />
<br />
backup1 is our primary customer backup system. As usage grows over time, it needs to be regularly purged of old files. The easiest way to do this is by removing deprecated files. These mostly consist of cancelled customers or temporary dump/storage files (created during dump/restores). Our standard policy is to hang onto cancelled customers for 6mos after which we remove their files (as far as customers know their data is purged immediately, but we hang onto it just in case.. and in some cases we cancel a server due to non payment so this makes it easy to restore their system). To find files to remove:<br />
<br />
<pre>[root@backup1 ~]# cd /data/deprecated/<br />
[root@backup1 /data/deprecated]# ls<br />
2101-migrated-20120317.tgz old-683-cxld-20121021.tgz<br />
69.55.230.2-wwwbackup old-744-cxld-20120708.tgz<br />
991-DONTDELETE.tgz old-809-cxld-20120609.tgz<br />
archive-col02050-mdfile-cxld-20120409.gz old-854-cxld-20120621.tgz<br />
col01371.tgz old-931-cxld-20060513.tgz<br />
deleteme_ubuntu-10.10-x86_20111205 old-col00123-mdfile-noarchive-20120417.gz<br />
jail10_old old-col00147-vnfile-cxld-20120828.gz<br />
jail14_rsync_old old-col00419-dump-cxld-20120224.gz<br />
jail15_old old-col01098-vnfile-cxld-20120827.gz<br />
jail3_old old-col01278-dump-cxld-20120822<br />
jail4_old old-col01517-dump-cxld-20120828<br />
jail5_old old-col01669-dump-cxld-20120203.gz<br />
old-1009-cxld-20120608.tgz old-col01687-dump-cxld-20120909<br />
old-1012-cxld-20120411.tgz old-col01790-dump-cxld-20120828<br />
old-1052-cxld-20120721.tgz old-col01812-dump-cxld-20120820<br />
old-10631-cxld-20120622.tgz old-col01938-mdfile-cxld-20120619.gz<br />
old-10632-cxld-20120622.tgz old-col02095-mdfile-noarchive-20120523.gz<br />
old-10633-cxld-20120622.tgz olddebian-3.0-v15-20110610.tgz<br />
old-1236-cxld-20120621.tgz oldmod_frontpage-deb30-v15-20110610.tgz<br />
old-1381-cxld-20120404.tgz oldmod_perl-deb30-v15-20110610.tgz<br />
old-1422-cxld-20120721.tgz oldmod_ssl-deb30-v15-20110610.tgz<br />
old-14681-cxld-20120619.tgz oldmysql-deb30-v15-20110610.tgz<br />
old-1544-cxld-20120626.tgz oldproftpd-deb30-v15-20110610.tgz<br />
old-18351-cxld-20120605.tgz old_virt14<br />
old-1853-cxld-20120910.tgz old_virt18<br />
old-1963-cxld-20120206.tgz oldwebmin-deb30-v15-20110610.tgz<br />
old-1967-cxld-20120605.tgz suse.virt11.20120421.tgz<br />
old-1981-noarchive-20120729.tgz virt11<br />
old-2030-migrated-noarchive-20120727.tgz virt12_old<br />
old-2037-cxld-20120716.tgz virt13_old<br />
old-2065-cxld-20120727.tgz virt16_old<br />
old-2068-cxld-20120424.tgz virt4_old<br />
old-2085-cxld-20120531.tgz virt5_old<br />
old-364-cxld-20120904.tgz virt6_old<br />
old-446-cxld-20120512.tgz virt7_old<br />
old-613-cxld-20120601.tgz virt8_old<br />
[root@backup1 /data/deprecated]#<br />
</pre><br />
<br />
virtX_old and jailX_old are permanently archived, so ignore those as well as anything else marked not to delete or otherwise suspicious. Likewise, probably a good idea to try to hang onto oldTEMPLATE.gz as long as we can as well.<br />
Most of the stuff we want to delete is dated when it was deprecated, making this easy. So to remove files from 6 mos ago (running this in Oct):<br />
<br />
<pre>[root@backup1 /data/deprecated]# ls old*201204*<br />
old-1012-cxld-20120411.tgz old-2068-cxld-20120424.tgz<br />
old-1381-cxld-20120404.tgz old-col00123-mdfile-noarchive-20120417.gz<br />
[root@backup1 /data/deprecated]# rm old*201204*<br />
</pre><br />
<br />
Every few months you will also want to remove some of the snapshot archives for mail. We typically save the 1st, 10th, and 20th of each month. To do this you set aside the dates you want to save then remove months at a time, followed by restoring the set aside dates. Here's how that works:<br />
<pre>[root@backup1 /data/www/daily]# ls<br />
05 08-10-11 10-04-10 11-10-10 12-07-29 12-09-21 12-11-14<br />
06 08-10-21 10-04-20 11-10-20 12-07-30 12-09-22 12-11-15<br />
06-06-01-usr-home.tgz 08-11-01 10-05-01 11-11-01 12-07-31 12-09-23 12-11-16<br />
06-07-01-usr-home.tgz 08-11-10 10-05-11 11-11-10 12-08-01 12-09-24 12-11-17<br />
06-08-01-usr-home.tgz 08-11-20 10-05-20 11-11-20 12-08-02 12-09-25 12-11-18<br />
06-09-01-usr-home.tgz 08-12-01 10-06-01 11-12-01 12-08-03 12-09-26 12-11-19<br />
06-11-10 08-12-10 10-06-10 11-12-10 12-08-04 12-09-27 12-11-20<br />
06-12-21 08-12-20 10-06-20 11-12-20 12-08-05 12-09-28 12-11-21<br />
07-01-10 09-01-01 10-07-01 12-01-01 12-08-06 12-09-29 12-11-22<br />
07-01-20 09-01-10 10-07-10 12-01-10 12-08-07 12-09-30 12-11-23<br />
07-02-10 09-01-20 10-07-20 12-01-20 12-08-08 12-10-01 12-11-24<br />
07-02-20 09-02-01 10-08-01 12-02-01 12-08-09 12-10-02 12-11-25<br />
07-03-01 09-02-10 10-08-10 12-02-10 12-08-10 12-10-03 12-11-26<br />
07-03-20 09-02-20 10-08-20 12-02-20 12-08-11 12-10-04 12-11-27<br />
07-04-01 09-03-01 10-09-01 12-03-01 12-08-12 12-10-05 12-11-28<br />
07-04-10 09-03-10 10-09-10 12-03-10 12-08-13 12-10-06 12-11-29<br />
07-04-20 09-03-20 10-09-20 12-03-20 12-08-14 12-10-07 12-11-30<br />
07-05-01 09-04-01 10-10-01 12-04-01 12-08-15 12-10-08 12-12-01<br />
07-05-10 09-04-10 10-10-10 12-04-10 12-08-16 12-10-09 12-12-02<br />
07-05-20 09-04-20 10-10-20 12-04-20 12-08-17 12-10-10 12-12-03<br />
07-06-01 09-05-01 10-11-01 12-05-01 12-08-18 12-10-11 12-12-04<br />
07-06-10 09-05-10 10-11-10 12-05-10 12-08-19 12-10-12 12-12-05<br />
07-06-20 09-05-20 10-11-20 12-05-20 12-08-20 12-10-13 12-12-06<br />
07-07-20 09-06-01 10-12-01 12-06-01 12-08-21 12-10-14 12-12-07<br />
07-08-10 09-06-10 10-12-10 12-06-10 12-08-22 12-10-15 12-12-08<br />
07-08-20 09-06-20 10-12-20 12-06-20 12-08-23 12-10-16 12-12-09<br />
07-09-01 09-07-01 11-01-01 12-07-01 12-08-24 12-10-17 12-12-10<br />
07-10-01 09-07-10 11-01-10 12-07-02 12-08-25 12-10-18 12-12-11<br />
07-10-10 09-07-20 11-01-21 12-07-03 12-08-26 12-10-19 12-12-12<br />
07-10-20 09-08-01 11-02-01 12-07-04 12-08-27 12-10-20 12-12-13<br />
07-12-01 09-08-10 11-02-10 12-07-05 12-08-28 12-10-21 12-12-14<br />
07-12-10 09-08-20 11-02-20 12-07-06 12-08-29 12-10-22 12-12-15<br />
08-01-01 09-09-01 11-03-01 12-07-07 12-08-30 12-10-23 12-12-16<br />
08-01-20 09-09-10 11-03-10 12-07-08 12-08-31 12-10-24 12-12-17<br />
08-02-20 09-09-20 11-03-20 12-07-09 12-09-01 12-10-25 12-12-18<br />
08-03-01 09-10-01 11-04-01 12-07-10 12-09-02 12-10-26 12-12-19<br />
08-03-10 09-10-10 11-04-10 12-07-11 12-09-03 12-10-27 12-12-20<br />
08-03-20 09-10-20 11-04-20 12-07-12 12-09-04 12-10-28 12-12-21<br />
08-04-01 09-11-01 11-05-01 12-07-13 12-09-05 12-10-29 12-12-22<br />
08-04-20 09-11-10 11-05-10 12-07-14 12-09-06 12-10-30 12-12-23<br />
08-05-01 09-11-20 11-05-20 12-07-15 12-09-07 12-10-31 12-12-24<br />
08-05-10 09-12-01 11-06-01 12-07-16 12-09-08 12-11-01 12-12-25<br />
08-06-10 09-12-10 11-06-10 12-07-17 12-09-09 12-11-02 12-12-26<br />
08-06-20 09-12-20 11-06-20 12-07-18 12-09-10 12-11-03 12-12-27<br />
08-07-02 10-01-01 11-07-01 12-07-19 12-09-11 12-11-04 12-12-28<br />
08-07-10 10-01-10 11-07-10 12-07-20 12-09-12 12-11-05 2008-10-23<br />
08-07-20 10-01-20 11-07-20 12-07-21 12-09-13 12-11-06 bb.tgz<br />
08-08-01 10-02-01 11-08-01 12-07-22 12-09-14 12-11-07 boot<br />
08-08-10 10-02-10 11-08-10 12-07-23 12-09-15 12-11-08 current<br />
08-08-21 10-02-20 11-08-20 12-07-24 12-09-16 12-11-09 hold<br />
08-09-01 10-03-01 11-09-01 12-07-25 12-09-17 12-11-10<br />
08-09-10 10-03-10 11-09-10 12-07-26 12-09-18 12-11-11<br />
08-09-21 10-03-20 11-09-20 12-07-27 12-09-19 12-11-12<br />
08-10-01 10-04-01 11-10-01 12-07-28 12-09-20 12-11-13<br />
[root@backup1 /data/www/daily]#<br />
</pre><br />
<br />
So we see that everything up to July 2012 has been pruned. To prune July 2012 we do the following:<br />
<br />
<pre>mv 12-07-01 hold<br />
mv 12-07-10 hold<br />
mv 12-07-20 hold<br />
rm -fr 12-07*<br />
mv hold/* .</pre></div>75.80.128.147https://wiki.jcihosting.com/index.php?title=Routine_Maintenance&diff=2201Routine Maintenance2020-02-25T17:50:43Z<p>75.80.128.147: /* check jail5 for crashed VPSs */</p>
<hr />
<div>= Daily Tasks =<br />
<br />
== check load graphs == <br />
<br />
Click on the Load link in mgmt <br />
<br />
This screen shows you load levels on our servers and network traffic for critical machines (firewalls, backup servers).<br />
<br />
If you see load high or increasing <br />
<br />
FreeBSD: <br />
run [[VPS_Management#jtop|jtop]] (or [[VPS_Management#jt|jt]] > 7.x) and see if there are any runaway processes. Here are some examples of entries in top that are<br />
definitely runaway processes:<br />
<br />
<pre>79481 root 64 0 2256K 1056K CPU1 1 58:16 87.40% 87.40% nano<br />
50650 1000 64 0 1852K 1112K RUN 0 207.9H 84.08% 84.08% screen<br />
14829 www 2 0 39100K 31736K accept 0 104:24 46.54% 6.54% httpd<br />
42065 root 61 0 1300K 844K RUN 1 47.8H 91.36% 91.36% ee<br />
1328 www 56 0 18440K 10796K CPU1 0 64.4H 97.71% 97.71% httpd<br />
26251 user 57 0 6124K 1160K CPU1 1 82.9H 98.44% 98.44% screen<br />
89874 root 60 0 1352K 892K RUN 1 33.8H 65.82% 65.82% dialog<br />
38656 1000 64 0 3088K 2136K CPU0 0 806:13 97.95% 97.95% StutBot<br />
27630 root 64 0 1396K 972K RUN 1 76.8H 86.47% 86.47% ee</pre><br />
<br />
Linux:<br />
run [[VPS_Management#vwe|vwe]] to see which VPS’s have high loads. From there run <tt>[[VPS_Management#vp|vp]] <veid></tt> and/or <tt>[[VPS_Management#vt|vt]] <veid></tt> to see what's going on in that system.<br />
[[VPS_Management#vzstat|vzstat]] will also give you a nice picture of whats going on, systems with high numbers in the mlat column are likely culprits.<br />
<br />
examples of out of control procs:<br />
<br />
<pre>12183 nobody 16 0 4916 1348 1340 R 45.5 0.0 4249m httpd<br />
29266 #502 16 0 1852 796 792 R 22.5 0.0 1104m vim<br />
23860 #41 16 0 5472 5472 2076 R 98.9 0.2 31:41 python<br />
19227 bin 19 0 1688 716 652 R 99.9 0.0 321:08 wtrs_ui<br />
7762 apache 16 0 268 236 224 R 85.7 0.0 1010m ptrace<br />
4624 #501 20 0 4304 2400 2044 R 53.6 0.1 284:32 YoSucker<br />
20451 #506 20 0 1876 820 816 R 17.2 0.0 169:35 vim<br />
8834 #514 20 0 900 724 672 R 77.6 0.0 382:30 neostats<br />
31815 apache 14 0 3176 3176 1696 R 74.4 0.1 6:15 counter</pre><br />
<br />
Just kill -9 them and be done with it.<br />
<br />
Also, anytime you see `kmod` or `ptrace` - kill those immediaely no matter how much they are using - they are attempts to exploit the linux ptrace bug. They won't work, but they suck a lot of CPU...<br />
<br />
Also, any other processes that are at 90-100% cpu usage and have been running for any long period of time should be killed except for mysqld processes on FreeBSD. See above.<br />
<br />
However, there is an exception:<br />
<br />
if it is a mysqld, we don't want to kill their database. What you want to do is <tt>[[VPS_Management#jpid|jpid]] <pid></tt> to see who owns it, and then email them the paste containing the instructions for the nanny. Or you can simply do a <tt>kill -1 PID</tt> on the process to restart it.<br />
<br />
=== Load averages jump at night ===<br />
<br />
The load averages on the FreeBSD systems may jump up at night between 1 and 4 am - this is because the backups are running - if this is what is causing a jump in load, you will see processes like `rsync` in top eating a lot of CPU time.<br />
<br />
== check backups ==<br />
<br />
mgmt -> Motnroing -> Backups and make sure every machine was backed up the previous nite. <br />
Also look at df on backup1 and backup2 to make sure no disk is approaching full, though bb should warn us in advance. Please note - errors encountered when a backup script on any of the particular systems run will gnerate an email to support@johncompanies.com so you can know immediately the day after if the directory to be backed up has been moved or no longer exists. A paste exists for this to notify the customer of a non-existant file/dir.<br />
<br />
== check bb for warnings ==<br />
mgmt -> BigBrother<br />
<br />
Some events don't generate pages (on purpose). You will only see them by going to the bb main page.<br />
<br />
== check jail5 for crashed VPSs ==<br />
<br />
On jail5<br />
<br />
notrunning<br />
<br />
To restart a VPS <br />
<br />
vm restart col0XXXX<br />
<br />
== check NetHere ==<br />
<br />
Check the NetHere servers<br />
<br />
=== Mail systems ===<br />
Check for possible SPAMMERs.<br />
<br />
==== Incoming ====<br />
Check mta-1 and mta-2 count of customer logins for possible<br />
customer SPAM compromises.<br />
<br />
login_count /logs/maillog | tail -30<br />
<br />
==== Outgoing ====<br />
<br />
Check outgoing queues on relay-1 and relay-2<br />
<br />
mail_count | tail -30<br />
<br />
=== Nagios ===<br />
<br />
Check for other problems on NetHere servers<br />
<br />
https://nagios.nethere.net<br />
<br />
= Monthly Tasks =<br />
== rotate pine sent mail (1st of month) ==<br />
On the 1st of the month, before any emails are sent out, quit out of pine, then log back in. Send mail from last month will be archived.<br />
If you mess up and do it on the 3rd (for example), you can go into the previous month's saved email and save emails from the current month into the <tt>sent-mail</tt> (current month) mailbox.<br />
<br />
== b/w caps ==<br />
On the 1st: remove any bwcaps put into the firewall (only really applies if a bwcap was added cause someone went over on b/w):<br />
<br />
ipfw list|grep pipe<br />
ipfw del [each rule listed]<br />
<br />
NOTE: this cronjob on newgateway will do some of that for you, provided you used one of the following pipe #s:<br />
0 0 1 * * /sbin/ipfw del 3 4 5 17331<br />
<br />
-------------------<br />
<br />
We really don’t do this anymore since we have centralized traffic accounting with netflow, but for posterity:<br />
<br />
Make sure all machines reset counters to 0 after midnight on the 1st<br />
Make sure they dumped a counter<br />
<br />
On each jail run:<br />
trafficgather.pl<br />
<br />
And on each virt:<br />
linuxtrafficgather.pl<br />
<br />
== Monthly RAID checks ==<br />
<br />
Every month we check the health of and verfy the parity on all our RAID-based systems.<br />
To facilitate this, we've created a simple script to start the process:<br />
<br />
sh /root/verify.sh<br />
<br />
=== Adaptec controllers ===<br />
Here's some sample output:<br />
<pre><br />
mail /usr/local/www/scripts# sh /root/verify.sh<br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> container list /f<br />
Executing: container list /full=TRUE<br />
Num Total Oth Chunk Scsi Partition<br />
Creation System<br />
Label Type Size Ctr Size Usage B:ID:L Offset:Size State RO Lk Task Done% Ent<br />
Date Time Files<br />
----- ------ ------ --- ------ ------- ------ ------------- ------- -- -- ------- ------ ---<br />
------ -------- ------<br />
0 Mirror 33.9GB Open 0:01:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:32<br />
/dev/aacd0 mirror0 0:00:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:32<br />
<br />
1 Mirror 33.9GB Open 0:02:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:50<br />
/dev/aacd1 mirror1 0:03:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:50<br />
<br />
<br />
AAC0> disk list /f<br />
Executing: disk list /full=TRUE<br />
<br />
B:ID:L Device Type Removable media Vendor-ID Product-ID Rev Blocks Bytes/Bl<br />
ock Usage Shared Rate<br />
------ -------------- --------------- --------- ---------------- ----- --------- --------<br />
--- ---------------- ------ ----<br />
0:00:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:01:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:02:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:03:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
<br />
AAC0> disk show smart<br />
Executing: disk show smart<br />
<br />
Smart Method of Enable<br />
Capable Informational Exception Performance Error<br />
B:ID:L Device Exceptions(MRIE) Control Enabled Count<br />
------ ------- ---------------- --------- ----------- ------<br />
0:00:0 Y 6 Y N 0<br />
0:01:0 Y 6 Y N 0<br />
0:02:0 Y 6 Y N 0<br />
0:03:0 Y 6 Y N 0<br />
0:06:0 N<br />
<br />
AAC0> task list<br />
Executing: task list<br />
<br />
Controller Tasks<br />
<br />
TaskId Function Done% Container State Specific1 Specific2<br />
------ -------- ------- --------- ----- --------- ---------<br />
<br />
No tasks currently running on controller<br />
<br />
AAC0> dia sh hi<br />
Executing: diagnostic show history<br />
No switches specified, defaulting to "/current".<br />
<br />
<br />
<br />
*** HISTORY BUFFER FROM CURRENT CONTROLLER RUN ***<br />
<br />
[00]: GetDiskLogEntry: container - 1, entry return 0<br />
[01]: Container 1 started SCRUB task<br />
[02]: Starting Mirror:1 scrub<br />
[03]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[04]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[05]: UpdateDiskLogIndex - Set - container 0, index 1<br />
[06]: GetDiskLogEntry: container - 0, entry return 1<br />
[07]: Container 0 started SCRUB task<br />
[08]: Starting Mirror:0 scrub<br />
[09]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[10]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[11]: Mirror Scrub Container:1 ErrorsFound:0<br />
[12]: Clear disk log: sector - 80, driveno 2<br />
[13]: Clear disk log: sector - 80, driveno 3<br />
[14]: Container 1 completed SCRUB task:<br />
[15]: Mirror Scrub Container:0 ErrorsFound:0<br />
[16]: Clear disk log: sector - 81, driveno 1<br />
[17]: Clear disk log: sector - 81, driveno 0<br />
[18]: Container 0 completed SCRUB task:<br />
[19]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[20]: GetDiskLogEntry: container - 0, entry return 0<br />
[21]: Container 0 started SCRUB task<br />
[22]: Starting Mirror:0 scrub<br />
[23]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[24]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[25]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[26]: GetDiskLogEntry: container - 1, entry return 1<br />
[27]: Container 1 started SCRUB task<br />
[28]: Starting Mirror:1 scrub<br />
[29]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[30]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[31]: Mirror Scrub Container:1 ErrorsFound:0<br />
[32]: Clear disk log: sector - 81, driveno 2<br />
[33]: Clear disk log: sector - 81, driveno 3<br />
[34]: Container 1 completed SCRUB task:<br />
[35]: Mirror Scrub Container:0 ErrorsFound:0<br />
[36]: Clear disk log: sector - 80, driveno 1<br />
[37]: Clear disk log: sector - 80, driveno 0<br />
[38]: Container 0 completed SCRUB task:<br />
[39]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[40]: GetDiskLogEntry: container - 0, entry return 0<br />
[41]: Container 0 started SCRUB task<br />
[42]: Starting Mirror:0 scrub<br />
[43]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[44]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[45]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[46]: GetDiskLogEntry: container - 1, entry return 1<br />
[47]: Container 1 started SCRUB task<br />
[48]: Starting Mirror:1 scrub<br />
[49]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[50]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[51]: Mirror Scrub Container:1 ErrorsFound:0<br />
[52]: Clear disk log: sector - 81, driveno 2<br />
[53]: Clear disk log: sector - 81, driveno 3<br />
[54]: Container 1 completed SCRUB task:<br />
[55]: Mirror Scrub Container:0 ErrorsFound:0<br />
[56]: Clear disk log: sector - 80, driveno 1<br />
[57]: Clear disk log: sector - 80, driveno 0<br />
[58]: Container 0 completed SCRUB task:<br />
[59]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[60]: GetDiskLogEntry: container - 0, entry return 0<br />
[61]: Container 0 started SCRUB task<br />
[62]: Starting Mirror:0 scrub<br />
[63]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[64]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[65]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[66]: GetDiskLogEntry: container - 1, entry return 1<br />
[67]: Container 1 started SCRUB task<br />
[68]: Starting Mirror:1 scrub<br />
[69]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[70]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[71]: Mirror Scrub Container:1 ErrorsFound:0<br />
[72]: Clear disk log: sector - 81, driveno 2<br />
[73]: Clear disk log: sector - 81, driveno 3<br />
[74]: Container 1 completed SCRUB task:<br />
[75]: Mirror Scrub Container:0 ErrorsFound:0<br />
[76]: Clear disk log: sector - 80, driveno 1<br />
[77]: Clear disk log: sector - 80, driveno 0<br />
[78]: Container 0 completed SCRUB task:<br />
[79]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[80]: GetDiskLogEntry: container - 0, entry return 0<br />
[81]: Container 0 started SCRUB task<br />
[82]: Starting Mirror:0 scrub<br />
[83]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[84]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[85]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[86]: GetDiskLogEntry: container - 1, entry return 1<br />
[87]: Container 1 started SCRUB task<br />
[88]: Starting Mirror:1 scrub<br />
[89]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[90]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[91]: Mirror Scrub Container:1 ErrorsFound:0<br />
[92]: Clear disk log: sector - 81, driveno 2<br />
[93]: Clear disk log: sector - 81, driveno 3<br />
[94]: Container 1 completed SCRUB task:<br />
[95]: Mirror Scrub Container:0 ErrorsFound:0<br />
[96]: Clear disk log: sector - 80, driveno 1<br />
[97]: Clear disk log: sector - 80, driveno 0<br />
[98]: Container 0 completed SCRUB task:<br />
[99]:<br />
<br />
========================<br />
History Output Complete.<br />
<br />
AAC0><br />
AAC0> exit<br />
Executing: exit<br />
<br />
press enter when ready to run verify <INS><br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> contai scr 0<br />
Executing: container scrub 0<br />
<br />
AAC0> contai scr 1<br />
Executing: container scrub 1<br />
<br />
AAC0> exit<br />
Executing: exit<br />
<br />
when done run: <br />
<br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
<br />
<br />
Nov 1 10:32:46 mail /kernel: aac0: **Monitor** Container 0 started SCRUB task<br />
Nov 1 10:32:47 mail /kernel: aac0: **Monitor** Container 1 started SCRUB task<br />
<br />
</pre><br />
<br />
Here's an analysis of what we're seeing and what we're looking for:<br />
<br />
<pre><br />
AAC0> container list /f<br />
Executing: container list /full=TRUE<br />
Num Total Oth Chunk Scsi Partition<br />
Creation System<br />
Label Type Size Ctr Size Usage B:ID:L Offset:Size State RO Lk Task Done% Ent<br />
Date Time Files<br />
----- ------ ------ --- ------ ------- ------ ------------- ------- -- -- ------- ------ ---<br />
------ -------- ------<br />
0 Mirror 33.9GB Open 0:01:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:32<br />
/dev/aacd0 mirror0 0:00:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:32<br />
<br />
1 Mirror 33.9GB Open 0:02:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:50<br />
/dev/aacd1 mirror1 0:03:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:50<br />
</pre><br />
<br />
This is showing you the health of the arrays. You're looking for ''Normal'' under the State column, and the absence of a ! in the sector size - sometimes, you'll see this:<br />
64.0KB!33.9GB <br />
<br />
That indicates a problem.<br />
<br />
<pre><br />
AAC0> disk show smart<br />
Executing: disk show smart<br />
<br />
Smart Method of Enable<br />
Capable Informational Exception Performance Error<br />
B:ID:L Device Exceptions(MRIE) Control Enabled Count<br />
------ ------- ---------------- --------- ----------- ------<br />
0:00:0 Y 6 Y N 0<br />
0:01:0 Y 6 Y N 0<br />
0:02:0 Y 6 Y N 0<br />
0:03:0 Y 6 Y N 0<br />
0:06:0 N<br />
</pre><br />
<br />
This shows you a SMART report output. Looking for values in the Error Count column.<br />
<pre><br />
AAC0> task list<br />
Executing: task list<br />
<br />
Controller Tasks<br />
<br />
TaskId Function Done% Container State Specific1 Specific2<br />
------ -------- ------- --------- ----- --------- ---------<br />
<br />
No tasks currently running on controller<br />
</pre><br />
<br />
Look for absence of tasks running- a bad thing would be to see a rebuild or verify running when you didn't initiate it.<br />
<br />
With the history output, you're looking for any anomalies or events since the last time a verify was run. If you see a drive with lots of problems, you may want to take backups before allowing the verify to run since it could replicate errors onto the good drive.<br />
<br />
After you see the history output, it will prompt you to press enter to run the verify. If you're happy with all the output you're seeing- mirror is healthy, history looks good, it's safe to proceed. Otherwise ^C to exit. After hitting enter it will start the verify and start to tail the messages log file (so you can easily see when the verify is complete). Here's what that'll look like:<br />
<br />
<pre><br />
Nov 1 14:38:08 mail /kernel: aac0: **Monitor** Container 1 completed SCRUB task:<br />
Nov 1 14:46:45 mail /kernel: aac0: **Monitor** Container 0 completed SCRUB task:<br />
</pre><br />
<br />
So, putting it all together, after hitting enter to start the verify, you'll see:<br />
<br />
<pre><br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> contai scr 0<br />
Executing: container scrub 0<br />
<br />
AAC0> contai scr 1<br />
Executing: container scrub 1<br />
<br />
AAC0> exit<br />
Executing: exit<br />
<br />
when done run: <br />
<br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
<br />
<br />
Nov 1 10:32:46 mail /kernel: aac0: **Monitor** Container 0 started SCRUB task<br />
Nov 1 10:32:47 mail /kernel: aac0: **Monitor** Container 1 started SCRUB task<br />
<br />
</pre><br />
<br />
When the scrub(s) (verify) are complete - if the server has multiple logical drives, it will run both in parallel - you should exit the tail of the log file (^C) and run:<br />
<br />
<pre><br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
</pre><br />
<br />
Which will show you the diagnostic history, you're looking for the results of the most recent scrub:<br />
<br />
<pre><br />
[100]: Mirror Scrub Container:1 ErrorsFound:0<br />
[101]: Clear disk log: sector - 81, driveno 2<br />
[102]: Clear disk log: sector - 81, driveno 3<br />
[103]: Container 1 completed SCRUB task:<br />
[104]: Mirror Scrub Container:0 ErrorsFound:0<br />
[105]: Clear disk log: sector - 80, driveno 1<br />
[106]: Clear disk log: sector - 80, driveno 0<br />
[107]: Container 0 completed SCRUB task:<br />
</pre><br />
<br />
^C to exit the RAID CLI.<br />
<br />
If you see:<br />
<br />
[104]: Mirror Scrub Container:0 ErrorsFound:5<br />
<br />
You'll want to rerun the verify on that drive till it shows 0, or perhaps replace the drive- you should be able to see from the output which drive had the problem.<br />
<br />
Depending on the size and how busy the drive is, the verify can take anywhere from an hour to the better part of a day.<br />
<br />
You will notice that the diagnostic history is not shown on our modern adaptec cards (i.e. any adaptec card not in a Dell 2450). The reason for this is the history is never cleared, so there's simply too much data to show and it just crashes the CLI. So, don't bother trying to see it...which does make it hard to see if there are problems going on, so you just need to watch the scrub to see it goes to 100%. You will also notice that on some servers there's no tail of messages. Again, this is cause no data is shown there about the completion of the scrub. The thing to do here is to go into the CLI and continue to <tt>show tasks</tt> to monitor scrub progress.<br />
<br />
See [[RAIC_CLI#Adaptec|Adaptec RAID CLI Reference]] for more details on how to use the CLI<br />
<br />
=== DELL (LSI-based) SAS controllers ===<br />
<br />
Here's what the output looks like when running verify.sh on a LSI-based card:<br />
<br />
<pre><br />
jail2 /mnt/data2# sh /root/verify.sh<br />
<br />
Adapter #0<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 0<br />
Device Id: 0<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018396142<br />
SAS Address(1): 0x0<br />
Connected Port Number: 0(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DESN<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 1<br />
Device Id: 1<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018395db2<br />
SAS Address(1): 0x0<br />
Connected Port Number: 1(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DERV<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 2<br />
Device Id: 2<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50006eece89<br />
SAS Address(1): 0x0<br />
Connected Port Number: 2(path0)<br />
Inquiry Data: SEAGATE ST3300555SS T2113LM4BFBZ<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 3<br />
Device Id: 3<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50006eee035<br />
SAS Address(1): 0x0<br />
Connected Port Number: 3(path0)<br />
Inquiry Data: SEAGATE ST3300555SS T2113LM4BGF7<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 4<br />
Device Id: 4<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50004bd7ea5<br />
SAS Address(1): 0x0<br />
Connected Port Number: 4(path0)<br />
Inquiry Data: SEAGATE ST3300656SS HS093QP0G8SW<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 5<br />
Device Id: 5<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e01f1c4112<br />
SAS Address(1): 0x0<br />
Connected Port Number: 5(path0)<br />
Inquiry Data: FUJITSU MBA3300RC D306BJ15P9201W06<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 0 (Target Id: 0)<br />
Name:<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:139392MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 1 (Target Id: 1)<br />
Name:MIRROR1<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 2 (Target Id: 2)<br />
Name:MIRROR2<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
Battery FRU : N/A<br />
Battery Warning : Enabled<br />
Memory Correctable Errors : 0<br />
Memory Uncorrectable Errors : 0<br />
BBU : Present<br />
BBU : Yes<br />
Cache When BBU Bad : Disabled<br />
press enter when ready to run verify<br />
<br />
</pre><br />
<br />
Before pressing enter, here's what we're looking for:<br />
<br />
<pre><br />
Enclosure Device ID: 32<br />
Slot Number: 0<br />
Device Id: 0<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018396142<br />
SAS Address(1): 0x0<br />
Connected Port Number: 0(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DESN<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
</pre><br />
<br />
This is the output shown for each physical drive in the system. We're looking to confirm it's Firmware state is ''Online'', and Media Error Count, Other Error Count, and Predictive Failure Count are all zero (or near zero).<br />
<br />
<pre><br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 1 (Target Id: 1)<br />
Name:MIRROR1<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
</pre><br />
<br />
This is the output for each logical drive. We're looking for State ''Optimal''. Also confirm <tt>Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU</tt><br />
<br />
<pre><br />
Exit Code: 0x00<br />
Battery FRU : N/A<br />
Battery Warning : Enabled<br />
Memory Correctable Errors : 0<br />
Memory Uncorrectable Errors : 0<br />
BBU : Present<br />
BBU : Yes<br />
Cache When BBU Bad : Disabled<br />
</pre><br />
<br />
Confirm that the battery is present and error-free.<br />
<br />
If all that checks out, you're ready to proceed with the verify. After pressing enter, the verify is started and here's what you see:<br />
<pre><br />
Start Check Consistency on Virtual Drive 0 (target id: 0) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Start Check Consistency on Virtual Drive 1 (target id: 1) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Start Check Consistency on Virtual Drive 2 (target id: 2) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Check Consistency<br />
<br />
Progress of Virtual Drives...<br />
<br />
Virtual Drive # Percent Complete Time Elps<br />
0 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:03<br />
1 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:02<br />
2 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:01<br />
<br />
Press <ESC> key to quit...<br />
</pre><br />
<br />
The progress for each drive is displayed until all drives have completed the verify. We just want to make sure that each drive goes to completion. No followup is needed...though there probably is a log or history where we can get more info.<br />
<br />
You will notice that jail7 does not run a verify- that's on purpose. The last time we tried this it crashed the system. So, this must be run from the BIOS (take the system offline for a couple hours).<br />
<br />
See [[RAIC_CLI#LSI|LSI RAID CLI Reference]] for more details on how to use the CLI<br />
<br />
=== LSI-based controllers (megaraid) ===<br />
<br />
There is a CLI for this however it's easier to do this with a curses GUI app: megaraid<br />
<br />
Currently only on these servers: virt15, virt16, and firewall2<br />
<br />
To run:<br />
<br />
# cd /usr/local/sbin/; megamgr<br />
<br />
Main menu:<br />
<pre>²ÚÄÄManagement MenuÄÄ¿²<br />
²³ Configure ³²<br />
²³ Initialize ³²<br />
²³ Objects ³²<br />
²³ Rebuild ³²<br />
²³ Check Consistency ³²<br />
²³ Advanced Menu ³²<br />
²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²</pre><br />
<br />
Before you check consistency, make sure the arrays are healthy.<br />
<br />
Objects -> Physical Drive<br />
<br />
Then look to make sure they're all <tt>ONLIN</tt><br />
<pre><br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄObjects - PHYSICAL DRIVE SELECTION MENUÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²<br />
²²²²²²²²³ ³²²²²²²<br />
²²²²²²²²³ Channel-1 ³²²²²²²<br />
²ÚÄÄMana³ ID ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»x ³²²²²²²<br />
²³ Confi³ 0º* ONLIN A01-01º ³²²²²²²<br />
²³ Initi³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²³ Objec³ 1º* ONLIN A01-02º ³²²²²²²<br />
²³ Rebui³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²³ Check³ 2º* ONLIN A02-01º ³²²²²²²<br />
²³ Advan³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²ÀÄÄÄÄÄij 3º* ONLIN A02-02º ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 4º* ONLIN A03-01º ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 5º* ONLIN A03-02ºþ ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 6º* º ³²²²²²²<br />
²²²²²²²²³ ÈÍÍÍÍÍÍÍÍÍÍÍÍÍͼx ³²²²²²²<br />
²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²<br />
²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²²²<br />
²²²²²²²²³Ch-1 ID-5 DISK 140013MB SEAGATE ST3146707LC 0003 ³²²²²²²²²<br />
²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²<br />
</pre><br />
<br />
<br />
Once that's done, hit escape once then the back arror to move back to the Objects menu.<br />
So you select Objects -> Logical Drive -> Logical Drive 1 -> Check Consistency -> YES<br />
<br />
<pre><br />
<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÚÄLogical Drives(02)Ä¿²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ Logical Drive 1 ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²ÚÄÄÄÄObjectsÄÄij Logical Drive 2 ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²ÚÄÄManagemen³ Adapter ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Configure ³ Logical Drive ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Initialize³ Physical Drive ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Objects ³ Channel ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Rebuild ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Check Consistency ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Advanced Menu ³²²²²²²²ÚÄÄÄÄLogical Drive 1ÄÄÄÄÄ¿²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²³ Initialize ÚÄCheck Consistency-1 ?Ä¿²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ Check Consiste³ YES ³²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ View/Update Pa³ NO ³²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³Select YES Or NO³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
</pre><br />
<br />
Then watch the progress. When done, escape back to Logical Drive then repeat for Logical Drive 2. If you ^C or accidentally escape out, you can come back in running the same commands and watch the progress again (it won't restart).<br />
<br />
You can exit megamgr by escaping out or ^C<br />
<br />
=== 3ware ===<br />
<br />
We are using 3ware controllers on backup1 & backup2. Running the verify script will give you different output based on the type of controller:<br />
<br />
<pre><br />
backup2 /d2# sh /root/verify.sh<br />
Controller: c0<br />
-------------<br />
Driver: 1.50.01.002<br />
Model: 7500-8<br />
FW: FE7X 1.05.00.068<br />
BIOS: BE7X 1.08.00.048<br />
Monitor: ME7X 1.01.00.040<br />
Serial #: F11605A3180172<br />
PCB: Rev3<br />
PCHIP: 1.30-33<br />
ACHIP: 3.20<br />
<br />
<br />
# of units: 3<br />
Unit 0: JBOD 186.31 GB ( 390721968 blocks): OK<br />
Unit 1: RAID 5 465.77 GB ( 976790016 blocks): DEGRADED<br />
Unit 5: RAID 5 698.65 GB ( 1465185024 blocks): DEGRADED<br />
<br />
# of ports: 8<br />
Port 0: WDC WD2000JB-00KFA0 WD-WCAMT1451690 186.31 GB (390721968 blocks): OK(unit 0)<br />
Port 1: WDC WD2500JB-00GVC0 WD-WCAL78219488 232.88 GB (488397168 blocks): OK(unit 1)<br />
Port 2: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT)<br />
Port 3: WDC WD2500JB-00GVC0 WD-WMAL73882417 232.88 GB (488397168 blocks): OK(unit 1)<br />
Port 4: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT)<br />
Port 5: WDC WD2500JB-00GVA0 WD-WMAL71338097 232.88 GB (488397168 blocks): OK(unit 5)<br />
Port 6: WDC WD2500JB-32EVA0 WD-WMAEH1301595 232.88 GB (488397168 blocks): OK(unit 5)<br />
Port 7: WDC WD2500JB-00GVC0 WD-WCAL78165566 232.88 GB (488397168 blocks): OK(unit 5)<br />
Controller: c1<br />
-------------<br />
Driver: 1.50.01.002<br />
Model: 7500-8<br />
FW: FE7X 1.05.00.068<br />
BIOS: BE7X 1.08.00.048<br />
Monitor: ME7X 1.01.00.040<br />
Serial #: F11605A3180167<br />
PCB: Rev3<br />
PCHIP: 1.30-33<br />
ACHIP: 3.20<br />
<br />
<br />
# of units: 2<br />
Unit 0: RAID 5 698.65 GB ( 1465185024 blocks): OK<br />
Unit 4: RAID 5 698.65 GB ( 1465185024 blocks): OK<br />
<br />
# of ports: 8<br />
Port 0: WDC WD2500JB-00GVA0 WD-WMAL71301258 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 1: WDC WD2500JB-00GVA0 WD-WMAL71322705 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 2: WDC WD2500JB-00GVA0 WD-WMAL71945050 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 3: WDC WD2500JB-00GVA0 WD-WMAL71316201 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 4: WDC WD2500JB-00GVC0 WD-WCAL78323749 232.88 GB (488397168 blocks): OK(unit 4)<br />
Port 5: WDC WD3200AAJB-00J3A0 WD-WCAV2V689068 298.09 GB (625142448 blocks): OK(unit 4)<br />
Port 6: WDC WD2500JB-00GVC0 WD-WCAL78234420 232.88 GB (488397168 blocks): OK(unit 4)<br />
Port 7: WDC WD2500JB-00GVC0 WD-WCAL78592213 232.88 GB (488397168 blocks): OK(unit 4)<br />
backup2 /d2#<br />
</pre><br />
<br />
On backup2 look for all ok, no verify.<br />
<br />
<pre><br />
[root@backup3 ~]# sh /root/verify.sh<br />
/c2 Driver Version = 1.26.02.002<br />
/c2 Model = 8006-2LP<br />
/c2 Available Memory = 512KB<br />
/c2 Firmware Version = FE8S 1.05.00.068<br />
/c2 Bios Version = BE7X 1.08.00.048<br />
/c2 Boot Loader Version = ME7X 1.01.00.040<br />
/c2 Serial Number = L018501C6481395<br />
/c2 PCB Version = Rev5<br />
/c2 PCHIP Version = 1.30-66<br />
/c2 ACHIP Version = 3.20<br />
/c2 Total Optimal Units = 1<br />
/c2 Not Optimal Units = 0<br />
<br />
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u1 RAID-1 OK - - - 931.512 ON -<br />
<br />
Port Status Unit Size Blocks Serial<br />
---------------------------------------------------------------<br />
p0 OK u1 931.51 GB 1953525168 WD-WMAW31148820<br />
p1 OK u1 931.51 GB 1953525168 WD-WCATR0277515<br />
<br />
<br />
Ctl Date Severity Alarm Message<br />
------------------------------------------------------------------------------<br />
<br />
Sending start verify message to /c2/u1 ... Done.<br />
<br />
<br />
when done run:<br />
tw_cli /c2 show alarms<br />
<br />
[root@backup3 ~]#<br />
</pre><br />
<br />
Automatically starts the verify, just run <tt>tw_cli /c2 show alarms</tt> as instructed to see the results of the verify.<br />
<br />
<pre><br />
[root@backup1 /data/deprecated]# sh /root/verify.sh<br />
/c0 Driver Version = 2.26.02.010<br />
/c0 Model = 9650SE-8LPML<br />
/c0 Available Memory = 224MB<br />
/c0 Firmware Version = FE9X 4.06.00.004<br />
/c0 Bios Version = BE9X 4.05.00.015<br />
/c0 Boot Loader Version = BL9X 3.08.00.001<br />
/c0 Serial Number = L326025A8270177<br />
/c0 PCB Version = Rev 032<br />
/c0 PCHIP Version = 2.00<br />
/c0 ACHIP Version = 1.90<br />
/c0 Number of Ports = 8<br />
/c0 Number of Drives = 6<br />
/c0 Number of Units = 1<br />
/c0 Total Optimal Units = 1<br />
/c0 Not Optimal Units = 0<br />
/c0 JBOD Export Policy = off<br />
/c0 Disk Spinup Policy = 1<br />
/c0 Spinup Stagger Time Policy (sec) = 1<br />
/c0 Auto-Carving Policy = off<br />
/c0 Auto-Carving Size = 2048 GB<br />
/c0 Auto-Rebuild Policy = on<br />
/c0 Controller Bus Type = PCIe<br />
/c0 Controller Bus Width = 1 lane<br />
/c0 Controller Bus Speed = 2.5 Gbps/lane<br />
<br />
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u0 RAID-5 OK - - 64K 4656.56 ON ON<br />
<br />
Port Status Unit Size Blocks Serial<br />
---------------------------------------------------------------<br />
p0 OK u0 931.51 GB 1953525168 9QJ1Y017<br />
p1 DEVICE-ERROR u0 931.51 GB 1953525168 9QJ1ZN07<br />
p2 OK u0 931.51 GB 1953525168 9QJ2XK1R<br />
p3 OK u0 931.51 GB 1953525168 9QJ2010B<br />
p4 OK u0 1.36 TB 2930277168 6XW0L36T<br />
p5 OK u0 931.51 GB 1953525168 WD-WMATV2444836<br />
p6 NOT-PRESENT - - - -<br />
p7 NOT-PRESENT - - - -<br />
<br />
<br />
Ctl Date Severity Alarm Message<br />
------------------------------------------------------------------------------<br />
c0 [Sat May 12 11:27:15 2012] WARNING Sector repair completed: port=0, LBA=0x6AE571C<br />
c0 [Sat May 12 19:16:21 2012] WARNING Sector repair completed: port=1, LBA=0x40E62A23<br />
c0 [Sat May 12 21:40:56 2012] INFO Verify completed: unit=0<br />
c0 [Mon May 14 00:53:53 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFA7<br />
c0 [Mon May 14 00:58:21 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAA<br />
c0 [Mon May 14 04:35:13 2012] WARNING Sector repair completed: port=0, LBA=0x8FEF2CF<br />
c0 [Mon May 14 04:38:22 2012] WARNING Sector repair completed: port=0, LBA=0x8FEF2D1<br />
c0 [Tue May 15 22:53:46 2012] WARNING Sector repair completed: port=0, LBA=0x13C2622<br />
c0 [Wed May 16 00:39:31 2012] WARNING Sector repair completed: port=0, LBA=0x365A67F<br />
c0 [Wed May 16 00:39:37 2012] WARNING Sector repair completed: port=0, LBA=0x365A685<br />
c0 [Wed May 16 00:47:18 2012] WARNING Sector repair completed: port=0, LBA=0x365A687<br />
c0 [Sat May 19 00:01:44 2012] INFO Verify started: unit=0<br />
c0 [Sat May 19 04:46:20 2012] WARNING Sector repair completed: port=0, LBA=0x365A68E<br />
c0 [Sat May 19 13:37:06 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAC<br />
c0 [Sat May 19 13:37:28 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAE<br />
c0 [Sat May 19 13:37:47 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFB1<br />
c0 [Sat May 19 13:38:00 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFB3<br />
c0 [Sat May 19 21:47:45 2012] INFO Verify completed: unit=0<br />
c0 [Wed May 23 12:21:41 2012] INFO Cache synchronization completed: unit=0<br />
c0 [Fri May 25 00:08:19 2012] WARNING Sector repair completed: port=0, LBA=0x12DA76C<br />
c0 [Fri May 25 00:08:34 2012] WARNING Sector repair completed: port=0, LBA=0x12E4901<br />
c0 [Fri May 25 00:09:33 2012] WARNING Sector repair completed: port=0, LBA=0x12DA773<br />
c0 [Fri May 25 00:39:12 2012] WARNING Sector repair completed: port=0, LBA=0x42C597B<br />
c0 [Sat May 26 00:01:45 2012] INFO Verify started: unit=0<br />
c0 [Sat May 26 00:42:05 2012] WARNING Sector repair completed: port=1, LBA=0x323C1AC<br />
c0 [Sat May 26 00:51:43 2012] WARNING Sector repair completed: port=1, LBA=0x323C1AE<br />
c0 [Sat May 26 01:54:21 2012] WARNING Sector repair completed: port=1, LBA=0x2F0D302<br />
c0 [Sat May 26 02:06:38 2012] WARNING Sector repair completed: port=0, LBA=0x12DA777<br />
c0 [Sat May 26 02:07:21 2012] WARNING Sector repair completed: port=0, LBA=0x12E48FE<br />
c0 [Sat May 26 04:20:00 2012] WARNING Sector repair completed: port=1, LBA=0x2F0D306<br />
c0 [Sat May 26 04:32:58 2012] WARNING Sector repair completed: port=1, LBA=0x323C1B1<br />
c0 [Sat May 26 04:33:21 2012] WARNING Sector repair completed: port=1, LBA=0x323C1B3<br />
c0 [Sat May 26 04:33:44 2012] WARNING Sector repair completed: port=1, LBA=0x323C1BA<br />
c0 [Sat May 26 05:24:07 2012] WARNING Sector repair completed: port=1, LBA=0x3F83862<br />
c0 [Sat May 26 05:25:09 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 06:08:13 2012] WARNING Sector repair completed: port=0, LBA=0x4CDC6A2<br />
c0 [Sat May 26 09:49:35 2012] WARNING Sector repair completed: port=1, LBA=0x6CACD4A<br />
c0 [Sat May 26 18:10:44 2012] WARNING Sector repair completed: port=1, LBA=0x18F425EA<br />
c0 [Sat May 26 19:45:40 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:22:52 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:15 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:22 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:35 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:41 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:49 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:57 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:24:02 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:54:41 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 22:00:30 2012] INFO Verify completed: unit=0<br />
c0 [Sat Jun 2 00:01:43 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 2 00:30:17 2012] WARNING Sector repair completed: port=0, LBA=0x2B911E4<br />
c0 [Sat Jun 2 00:50:57 2012] WARNING Sector repair completed: port=0, LBA=0x5A807CA6<br />
c0 [Sat Jun 2 04:13:13 2012] WARNING Sector repair completed: port=0, LBA=0x2D18291<br />
c0 [Sat Jun 2 04:13:35 2012] WARNING Sector repair completed: port=0, LBA=0x2D1829F<br />
c0 [Sat Jun 2 21:48:02 2012] INFO Verify completed: unit=0<br />
c0 [Mon Jun 4 04:40:34 2012] WARNING Sector repair completed: port=1, LBA=0x4AF8098F<br />
c0 [Tue Jun 5 00:28:19 2012] WARNING Sector repair completed: port=1, LBA=0x263C5CD<br />
c0 [Tue Jun 5 00:33:06 2012] WARNING Sector repair completed: port=1, LBA=0x263C5CF<br />
c0 [Thu Jun 7 00:34:27 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B5F<br />
c0 [Thu Jun 7 00:38:50 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B61<br />
c0 [Fri Jun 8 00:07:13 2012] WARNING Sector repair completed: port=0, LBA=0xC131F6B<br />
c0 [Sat Jun 9 00:01:41 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 9 00:29:11 2012] WARNING Sector repair completed: port=0, LBA=0x6C7614D<br />
c0 [Sat Jun 9 00:38:25 2012] WARNING Sector repair completed: port=0, LBA=0x6C76152<br />
c0 [Sat Jun 9 04:02:30 2012] WARNING Sector repair completed: port=1, LBA=0x263C5D1<br />
c0 [Sat Jun 9 04:02:52 2012] WARNING Sector repair completed: port=1, LBA=0x263C5D3<br />
c0 [Sat Jun 9 04:07:32 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E12<br />
c0 [Sat Jun 9 04:07:57 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E15<br />
c0 [Sat Jun 9 04:08:16 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E17<br />
c0 [Sat Jun 9 04:08:45 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E19<br />
c0 [Sat Jun 9 04:15:04 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B64<br />
c0 [Sat Jun 9 04:15:26 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B66<br />
c0 [Sat Jun 9 04:15:45 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B68<br />
c0 [Sat Jun 9 04:15:59 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B6C<br />
c0 [Sat Jun 9 04:16:13 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B6E<br />
c0 [Sat Jun 9 21:48:52 2012] INFO Verify completed: unit=0<br />
c0 [Thu Jun 14 00:40:10 2012] WARNING Sector repair completed: port=0, LBA=0x334F14B<br />
c0 [Sat Jun 16 00:01:38 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 16 21:16:19 2012] INFO Verify completed: unit=0<br />
c0 [Tue Jun 19 02:03:43 2012] WARNING Sector repair completed: port=1, LBA=0xFE41EAD<br />
c0 [Wed Jun 20 02:30:02 2012] WARNING Sector repair completed: port=1, LBA=0xD99145C<br />
c0 [Sat Jun 23 00:01:36 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 23 04:27:04 2012] WARNING Sector repair completed: port=1, LBA=0x2FAD311<br />
c0 [Sat Jun 23 06:52:38 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC8D<br />
c0 [Sat Jun 23 06:53:03 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC91<br />
c0 [Sat Jun 23 06:53:21 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC94<br />
c0 [Sat Jun 23 17:00:22 2012] WARNING Sector repair completed: port=1, LBA=0xF9AC7C9<br />
c0 [Sat Jun 23 21:15:19 2012] INFO Verify completed: unit=0<br />
c0 [Sat Jun 30 00:01:34 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 30 05:24:13 2012] WARNING Sector repair completed: port=0, LBA=0x3FAA9E7<br />
c0 [Sat Jun 30 14:49:39 2012] WARNING Sector repair completed: port=1, LBA=0x869931C<br />
c0 [Sat Jun 30 21:31:05 2012] INFO Verify completed: unit=0<br />
c0 [Tue Jul 3 03:40:25 2012] WARNING Sector repair completed: port=1, LBA=0xD36C7F7<br />
c0 [Fri Jul 6 02:50:18 2012] WARNING Sector repair completed: port=1, LBA=0x3562470<br />
c0 [Fri Jul 6 22:18:26 2012] WARNING Sector repair completed: port=1, LBA=0x3563173<br />
c0 [Sat Jul 7 00:01:31 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 7 00:50:16 2012] WARNING Sector repair completed: port=0, LBA=0x76EE88<br />
c0 [Sat Jul 7 00:50:39 2012] WARNING Sector repair completed: port=0, LBA=0x76EE8F<br />
c0 [Sat Jul 7 21:39:36 2012] INFO Verify completed: unit=0<br />
c0 [Sun Jul 8 02:51:05 2012] WARNING Sector repair completed: port=0, LBA=0x67759D<br />
c0 [Sun Jul 8 02:53:55 2012] WARNING Sector repair completed: port=0, LBA=0x67759B<br />
c0 [Tue Jul 10 16:17:21 2012] WARNING Sector repair completed: port=0, LBA=0x15C8C695<br />
c0 [Wed Jul 11 22:51:22 2012] WARNING Sector repair completed: port=1, LBA=0x355BBD0<br />
c0 [Sat Jul 14 00:01:28 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 14 01:33:40 2012] WARNING Sector repair completed: port=1, LBA=0x1333BCF4<br />
c0 [Sat Jul 14 03:36:23 2012] WARNING Sector repair completed: port=1, LBA=0x2174773<br />
c0 [Sat Jul 14 11:26:44 2012] WARNING Sector repair completed: port=0, LBA=0x7429AB7<br />
c0 [Sat Jul 14 16:53:50 2012] WARNING Sector repair completed: port=1, LBA=0xA17EB3F<br />
c0 [Sat Jul 14 21:19:25 2012] INFO Verify completed: unit=0<br />
c0 [Wed Jul 18 05:08:47 2012] WARNING Sector repair completed: port=1, LBA=0x17D62EDC<br />
c0 [Wed Jul 18 05:14:15 2012] WARNING Sector repair completed: port=1, LBA=0x17D62EE1<br />
c0 [Thu Jul 19 03:24:59 2012] WARNING Sector repair completed: port=0, LBA=0x7733C3D<br />
c0 [Thu Jul 19 03:25:20 2012] WARNING Sector repair completed: port=0, LBA=0x773CEA5<br />
c0 [Thu Jul 19 03:28:16 2012] WARNING Sector repair completed: port=0, LBA=0x7733C42<br />
c0 [Thu Jul 19 03:28:41 2012] WARNING Sector repair completed: port=0, LBA=0x773CEAF<br />
c0 [Sat Jul 21 00:01:26 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 21 03:07:30 2012] WARNING Sector repair completed: port=1, LBA=0x1CC6936<br />
c0 [Sat Jul 21 03:07:52 2012] WARNING Sector repair completed: port=1, LBA=0x1CC6938<br />
c0 [Sat Jul 21 03:08:11 2012] WARNING Sector repair completed: port=1, LBA=0x1CC693A<br />
c0 [Sat Jul 21 16:43:56 2012] WARNING Sector repair completed: port=0, LBA=0xD04C914<br />
c0 [Sat Jul 21 16:45:31 2012] WARNING Sector repair completed: port=1, LBA=0xD456973<br />
c0 [Sat Jul 21 21:14:29 2012] INFO Verify completed: unit=0<br />
c0 [Wed Jul 25 03:37:25 2012] WARNING Sector repair completed: port=0, LBA=0x1F8E6C43<br />
c0 [Sat Jul 28 00:01:24 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 28 01:45:27 2012] WARNING Sector repair completed: port=0, LBA=0x11584AD<br />
c0 [Sat Jul 28 18:54:25 2012] WARNING Sector repair completed: port=1, LBA=0x447C3E6C<br />
c0 [Sat Jul 28 21:13:46 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 1 03:20:11 2012] WARNING Sector repair completed: port=0, LBA=0x805FEF<br />
c0 [Fri Aug 3 00:50:03 2012] WARNING Sector repair completed: port=0, LBA=0xCED0ACA<br />
c0 [Sat Aug 4 00:01:22 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 4 00:52:51 2012] WARNING Sector repair completed: port=0, LBA=0x805FF3<br />
c0 [Sat Aug 4 00:53:14 2012] WARNING Sector repair completed: port=0, LBA=0x805FF5<br />
c0 [Sat Aug 4 00:53:33 2012] WARNING Sector repair completed: port=0, LBA=0x805FF7<br />
c0 [Sat Aug 4 00:53:47 2012] WARNING Sector repair completed: port=0, LBA=0x805FF9<br />
c0 [Sat Aug 4 00:54:00 2012] WARNING Sector repair completed: port=0, LBA=0x805FFB<br />
c0 [Sat Aug 4 00:54:14 2012] WARNING Sector repair completed: port=0, LBA=0x805FFD<br />
c0 [Sat Aug 4 00:54:27 2012] WARNING Sector repair completed: port=0, LBA=0x805FFF<br />
c0 [Sat Aug 4 04:43:12 2012] WARNING Sector repair completed: port=1, LBA=0x16974289<br />
c0 [Sat Aug 4 04:58:17 2012] WARNING Sector repair completed: port=1, LBA=0x1697428E<br />
c0 [Sat Aug 4 20:54:53 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 8 03:21:55 2012] ERROR Drive timeout detected: port=1<br />
c0 [Wed Aug 8 15:31:44 2012] WARNING Sector repair completed: port=0, LBA=0x1A366CD3<br />
c0 [Sat Aug 11 00:01:21 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 11 20:40:51 2012] INFO Verify completed: unit=0<br />
c0 [Thu Aug 16 05:10:55 2012] WARNING Sector repair completed: port=0, LBA=0x1C22593<br />
c0 [Sat Aug 18 00:01:18 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 18 03:00:20 2012] WARNING Sector repair completed: port=0, LBA=0x1C225A5<br />
c0 [Sat Aug 18 03:43:00 2012] WARNING Sector repair completed: port=1, LBA=0x23EE91E<br />
c0 [Sat Aug 18 03:43:23 2012] WARNING Sector repair completed: port=1, LBA=0x23EE920<br />
c0 [Sat Aug 18 17:00:06 2012] WARNING Sector repair completed: port=1, LBA=0x137D066A<br />
c0 [Sat Aug 18 17:00:29 2012] WARNING Sector repair completed: port=1, LBA=0x137D066D<br />
c0 [Sat Aug 18 21:13:01 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 22 01:36:08 2012] WARNING Sector repair completed: port=0, LBA=0x2560A0F<br />
c0 [Wed Aug 22 01:37:42 2012] WARNING Sector repair completed: port=0, LBA=0x2560A13<br />
c0 [Fri Aug 24 04:01:36 2012] WARNING Sector repair completed: port=1, LBA=0x55C1A5DF<br />
c0 [Fri Aug 24 05:02:06 2012] WARNING Sector repair completed: port=1, LBA=0xCE3378A<br />
c0 [Sat Aug 25 00:01:17 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 25 00:31:06 2012] WARNING Sector repair completed: port=1, LBA=0x50F65D<br />
c0 [Sat Aug 25 00:39:52 2012] WARNING Sector repair completed: port=0, LBA=0x678FF4<br />
c0 [Sat Aug 25 03:43:15 2012] WARNING Sector repair completed: port=0, LBA=0x2560A15<br />
c0 [Sat Aug 25 03:43:39 2012] WARNING Sector repair completed: port=0, LBA=0x2560A19<br />
c0 [Sat Aug 25 03:43:58 2012] WARNING Sector repair completed: port=0, LBA=0x2560A1B<br />
c0 [Sat Aug 25 03:44:30 2012] WARNING Sector repair completed: port=0, LBA=0x2560A21<br />
c0 [Sat Aug 25 20:58:14 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 29 04:57:15 2012] WARNING Sector repair completed: port=1, LBA=0xF3957EB<br />
c0 [Sat Sep 1 00:01:15 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 1 03:21:52 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC86<br />
c0 [Sat Sep 1 03:22:15 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC88<br />
c0 [Sat Sep 1 03:22:34 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC8A<br />
c0 [Sat Sep 1 03:22:47 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC8C<br />
c0 [Sat Sep 1 17:17:22 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD1<br />
c0 [Sat Sep 1 17:17:45 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD3<br />
c0 [Sat Sep 1 17:18:04 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD5<br />
c0 [Sat Sep 1 21:36:56 2012] INFO Verify completed: unit=0<br />
c0 [Thu Sep 6 00:07:30 2012] WARNING Sector repair completed: port=0, LBA=0xDA3C64B<br />
c0 [Thu Sep 6 00:32:56 2012] WARNING Sector repair completed: port=1, LBA=0x6BBA816<br />
c0 [Sat Sep 8 00:01:13 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 8 00:09:56 2012] WARNING Sector repair completed: port=0, LBA=0xDEBC958<br />
c0 [Sat Sep 8 04:38:45 2012] WARNING Sector repair completed: port=0, LBA=0x38D254F<br />
c0 [Sat Sep 8 20:44:50 2012] INFO Verify completed: unit=0<br />
c0 [Mon Sep 10 01:26:34 2012] WARNING Sector repair completed: port=1, LBA=0xFFD8D5E<br />
c0 [Wed Sep 12 00:33:48 2012] WARNING Sector repair completed: port=1, LBA=0xE8DB928<br />
c0 [Wed Sep 12 00:36:33 2012] WARNING Sector repair completed: port=1, LBA=0x6D49411<br />
c0 [Fri Sep 14 01:59:39 2012] WARNING Sector repair completed: port=0, LBA=0x1467F1C<br />
c0 [Fri Sep 14 02:08:27 2012] WARNING Sector repair completed: port=1, LBA=0x14C8ABD<br />
c0 [Fri Sep 14 03:54:47 2012] WARNING Sector repair completed: port=0, LBA=0x1580C915<br />
c0 [Sat Sep 15 00:01:11 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 15 02:38:14 2012] WARNING Sector repair completed: port=0, LBA=0x1C178973<br />
c0 [Sat Sep 15 02:59:02 2012] WARNING Sector repair completed: port=0, LBA=0x1C178975<br />
c0 [Sat Sep 15 04:47:08 2012] WARNING Sector repair completed: port=0, LBA=0x3FA0356<br />
c0 [Sat Sep 15 04:47:31 2012] WARNING Sector repair completed: port=0, LBA=0x3FA0359<br />
c0 [Sat Sep 15 10:41:59 2012] WARNING Sector repair completed: port=0, LBA=0x6DFD1EC<br />
c0 [Sat Sep 15 13:25:23 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD100<br />
c0 [Sat Sep 15 13:25:31 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD104<br />
c0 [Sat Sep 15 13:25:54 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD106<br />
c0 [Sat Sep 15 17:10:50 2012] WARNING Sector repair completed: port=0, LBA=0x1C178977<br />
c0 [Sat Sep 15 20:59:57 2012] INFO Verify completed: unit=0<br />
c0 [Tue Sep 18 01:17:18 2012] WARNING Sector repair completed: port=1, LBA=0x803B05B<br />
c0 [Sat Sep 22 00:01:10 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 22 20:54:31 2012] INFO Verify completed: unit=0<br />
c0 [Tue Sep 25 01:56:47 2012] WARNING Sector repair completed: port=0, LBA=0x26E3909<br />
c0 [Sat Sep 29 00:01:08 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 29 02:04:14 2012] WARNING Sector repair completed: port=0, LBA=0x146AC03<br />
c0 [Sat Sep 29 10:58:39 2012] WARNING Sector repair completed: port=0, LBA=0x6D4EB0E<br />
c0 [Sat Sep 29 10:59:02 2012] WARNING Sector repair completed: port=0, LBA=0x6D4EB14<br />
c0 [Sat Sep 29 11:22:44 2012] WARNING Sector repair completed: port=0, LBA=0x6F79623<br />
c0 [Sat Sep 29 13:50:48 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D65E<br />
c0 [Sat Sep 29 13:51:11 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D661<br />
c0 [Sat Sep 29 13:51:30 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D663<br />
c0 [Sat Sep 29 20:57:34 2012] INFO Verify completed: unit=0<br />
c0 [Mon Oct 1 04:47:24 2012] WARNING Sector repair completed: port=0, LBA=0xC5BC6F2<br />
c0 [Tue Oct 2 02:00:27 2012] WARNING Sector repair completed: port=0, LBA=0x1547667<br />
c0 [Tue Oct 2 02:01:56 2012] WARNING Sector repair completed: port=0, LBA=0x154766F<br />
c0 [Tue Oct 2 05:02:31 2012] WARNING Sector repair completed: port=1, LBA=0xD67D054<br />
c0 [Tue Oct 2 05:04:14 2012] WARNING Sector repair completed: port=1, LBA=0xD67D056<br />
c0 [Wed Oct 3 01:22:12 2012] WARNING Sector repair completed: port=1, LBA=0x12AAF8CA<br />
c0 [Thu Oct 4 04:29:22 2012] WARNING Sector repair completed: port=0, LBA=0x13E6F992<br />
c0 [Thu Oct 4 05:10:51 2012] WARNING Sector repair completed: port=0, LBA=0x1C252A4<br />
c0 [Sat Oct 6 00:01:07 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 6 19:41:18 2012] WARNING Sector repair completed: port=1, LBA=0x5A5C3AE8<br />
c0 [Sat Oct 6 21:01:05 2012] INFO Verify completed: unit=0<br />
c0 [Mon Oct 8 00:32:06 2012] WARNING Sector repair completed: port=0, LBA=0x6C60D3E<br />
c0 [Tue Oct 9 03:51:03 2012] WARNING Sector repair completed: port=1, LBA=0x89B5EC9<br />
c0 [Thu Oct 11 04:21:17 2012] WARNING Sector repair completed: port=1, LBA=0x13F85833<br />
c0 [Sat Oct 13 00:01:05 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 13 05:12:40 2012] WARNING Sector repair completed: port=0, LBA=0x3FA5134<br />
c0 [Sat Oct 13 21:08:35 2012] INFO Verify completed: unit=0<br />
c0 [Tue Oct 16 03:53:50 2012] WARNING Sector repair completed: port=1, LBA=0x148AA1BD<br />
c0 [Thu Oct 18 03:20:30 2012] WARNING Sector repair completed: port=0, LBA=0x1C8DABCB<br />
c0 [Thu Oct 18 04:52:50 2012] WARNING Sector repair completed: port=0, LBA=0xE879057<br />
c0 [Sat Oct 20 00:01:04 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 20 02:19:25 2012] WARNING Sector repair completed: port=1, LBA=0x174B012<br />
c0 [Sat Oct 20 03:41:38 2012] WARNING Sector repair completed: port=0, LBA=0x256D93B<br />
c0 [Sat Oct 20 03:42:01 2012] WARNING Sector repair completed: port=0, LBA=0x256D93D<br />
c0 [Sat Oct 20 03:42:40 2012] WARNING Sector repair completed: port=0, LBA=0x256D940<br />
c0 [Sat Oct 20 03:42:59 2012] WARNING Sector repair completed: port=0, LBA=0x256D942<br />
c0 [Sat Oct 20 03:43:12 2012] WARNING Sector repair completed: port=0, LBA=0x256D944<br />
c0 [Sat Oct 20 03:43:26 2012] WARNING Sector repair completed: port=0, LBA=0x256D948<br />
c0 [Sat Oct 20 16:37:52 2012] WARNING Sector repair completed: port=0, LBA=0xE879060<br />
c0 [Sat Oct 20 16:38:15 2012] WARNING Sector repair completed: port=0, LBA=0xE879062<br />
c0 [Sat Oct 20 21:00:18 2012] INFO Verify completed: unit=0<br />
c0 [Sat Oct 20 23:49:01 2012] WARNING Sector repair completed: port=1, LBA=0x4473E908<br />
c0 [Sun Oct 21 03:42:26 2012] WARNING Sector repair completed: port=1, LBA=0x175BADD5<br />
c0 [Tue Oct 23 01:09:04 2012] WARNING Sector repair completed: port=1, LBA=0x6E524860<br />
c0 [Fri Oct 26 03:21:25 2012] WARNING Sector repair completed: port=0, LBA=0x802C61<br />
c0 [Fri Oct 26 04:22:21 2012] WARNING Sector repair completed: port=0, LBA=0x176353CD<br />
c0 [Sat Oct 27 00:01:03 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 27 00:49:35 2012] WARNING Sector repair completed: port=0, LBA=0x802C65<br />
c0 [Sat Oct 27 17:02:24 2012] WARNING Sector repair completed: port=1, LBA=0xC1FF26D<br />
c0 [Sat Oct 27 17:09:06 2012] WARNING Sector repair completed: port=0, LBA=0xDF621AD<br />
c0 [Sat Oct 27 21:30:57 2012] INFO Verify completed: unit=0<br />
c0 [Tue Oct 30 00:20:46 2012] WARNING Sector repair completed: port=0, LBA=0xE9FE2AB<br />
c0 [Wed Oct 31 02:02:03 2012] WARNING Sector repair completed: port=0, LBA=0x1460C25<br />
c0 [Wed Oct 31 02:04:05 2012] WARNING Sector repair completed: port=0, LBA=0x1460C28<br />
c0 [Thu Nov 1 00:48:34 2012] WARNING Sector repair completed: port=1, LBA=0xA7C92BE<br />
c0 [Thu Nov 1 05:04:45 2012] WARNING Sector repair completed: port=0, LBA=0x1C252C2<br />
<br />
[root@backup1 /data/deprecated]#<br />
</pre><br />
<br />
Look for failed drives and errors. Obviously from the above we need to probably replace drives 0 and 1 and drive 1 is even showing as having problems, yet the RAID array is healthy, amazingly. You also see the automatic verifies.<br />
<br />
Note: when rebuilding a degraded mirror, you will see no progress as it rebuilds in the cli<br />
<pre>Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u0 RAID-5 REBUILDING 0 - 64K 4656.56 OFF ON</pre><br />
<br />
<br />
=== areca ===<br />
<br />
We are using an areca controller on backup3. <br />
<br />
<pre><br />
[root@newbackup3 ~]# sh /root/verify.sh<br />
# Name Raid Name Level Capacity Ch/Id/Lun State<br />
===============================================================================<br />
1 ARC-1160-VOL#00 Raid Set # 00 Raid5 5000.0GB 00/00/00 Checking(19.7%)<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
# Name Disks TotalCap FreeCap DiskChannels State<br />
===============================================================================<br />
1 Raid Set # 00 6 6000.0GB 0.0GB 123456 Checking<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
Date-Time Device Event Type Elapsed Time Errors<br />
===============================================================================<br />
2012-12-05 20:40:58 ARC-1160-VOL#00 Start Checking<br />
2012-12-01 05:06:04 ARC-1160-VOL#00 Complete Init 027:30:45<br />
2012-11-30 01:35:19 ARC-1160-VOL#00 Start Initialize<br />
2026-08-06 01:34:52 H/W MONITOR Raid Powered On<br />
2012-11-30 01:33:36 ARC-1160-VOL#00 Stop Initialization 000:31:48<br />
2012-11-30 01:01:47 ARC-1160-VOL#00 Start Initialize<br />
2026-08-06 00:58:13 H/W MONITOR Raid Powered On<br />
2012-11-30 00:57:26 ARC-1160-VOL#00 Stop Initialization 000:57:07<br />
2012-11-30 00:00:19 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 23:56:48 H/W MONITOR Raid Powered On<br />
2026-08-05 23:52:58 H/W MONITOR Raid Powered On<br />
2026-08-05 23:50:14 H/W MONITOR Raid Powered On<br />
2026-08-05 23:43:30 H/W MONITOR Raid Powered On<br />
2012-11-29 23:10:07 ARC-1160-VOL#00 Stop Initialization 000:00:56<br />
2012-11-29 23:09:11 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 23:08:57 H/W MONITOR Raid Powered On<br />
2012-11-29 23:08:10 ARC-1160-VOL#00 Stop Initialization 000:20:41<br />
2012-11-29 22:47:29 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 22:46:59 H/W MONITOR Raid Powered On<br />
2026-08-05 22:45:55 H/W MONITOR Raid Powered On<br />
2026-08-05 22:44:53 H/W MONITOR Raid Powered On<br />
2026-08-05 22:42:06 H/W MONITOR Raid Powered On<br />
2026-08-05 22:40:50 H/W MONITOR Raid Powered On<br />
2012-11-29 22:40:04 ARC-1160-VOL#00 Stop Initialization 000:24:25<br />
2012-11-29 22:15:38 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 22:15:11 000:000001215B00 Restart Init LBA Point<br />
2026-08-05 22:15:10 H/W MONITOR Raid Powered On<br />
2012-11-29 21:56:38 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 21:56:12 H/W MONITOR Raid Powered On<br />
2026-08-05 21:56:04 IDE Channel #03 Device Inserted<br />
2012-11-29 21:55:13 IDE Channel #04 Device Inserted<br />
2012-11-29 21:55:03 IDE Channel #02 Device Inserted<br />
2026-08-05 21:53:09 H/W MONITOR Raid Powered On<br />
2026-08-05 20:51:46 H/W MONITOR Raid Powered On<br />
2026-08-05 20:49:56 H/W MONITOR Raid Powered On<br />
2026-08-05 20:48:29 H/W MONITOR Raid Powered On<br />
2026-08-05 20:46:29 H/W MONITOR Raid Powered On<br />
2026-08-05 20:44:49 H/W MONITOR Raid Powered On<br />
2026-08-05 20:43:01 H/W MONITOR Raid Powered On<br />
2026-08-05 20:36:25 H/W MONITOR Raid Powered On<br />
2026-08-05 20:31:18 H/W MONITOR Raid Powered On<br />
2026-08-05 20:30:08 H/W MONITOR Raid Powered On<br />
2026-08-05 20:08:40 H/W MONITOR Raid Powered On<br />
2026-08-05 20:06:11 H/W MONITOR Raid Powered On<br />
2026-08-05 20:05:14 H/W MONITOR Raid Powered On<br />
2026-08-05 20:03:58 H/W MONITOR Raid Powered On<br />
2026-08-05 20:00:56 H/W MONITOR Raid Powered On<br />
2026-08-05 19:57:57 H/W MONITOR Raid Powered On<br />
2026-08-05 19:56:15 H/W MONITOR Raid Powered On<br />
2026-08-05 19:55:05 H/W MONITOR Raid Powered On<br />
2026-08-05 17:24:36 H/W MONITOR Raid Powered On<br />
2026-08-05 17:22:43 H/W MONITOR Raid Powered On<br />
2026-08-05 04:50:42 H/W MONITOR Raid Powered On<br />
2026-08-05 04:47:33 H/W MONITOR Raid Powered On<br />
2026-08-05 04:43:57 H/W MONITOR Raid Powered On<br />
2026-08-05 04:18:52 H/W MONITOR Raid Powered On<br />
2026-08-05 04:17:30 H/W MONITOR Raid Powered On<br />
2026-08-05 04:13:30 H/W MONITOR Raid Powered On<br />
2026-08-05 04:10:26 H/W MONITOR Raid Powered On<br />
2026-08-05 04:09:23 H/W MONITOR Raid Powered On<br />
2026-08-05 00:08:09 H/W MONITOR Raid Powered On<br />
2026-08-05 00:07:12 H/W MONITOR Raid Powered On<br />
2026-08-05 00:05:51 H/W MONITOR Raid Powered On<br />
2026-08-05 00:04:27 H/W MONITOR Raid Powered On<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
press enter when ready to run verify<br />
</pre><br />
<br />
Look for failed drives and errors. <br />
<br />
When it proceed's to verifying, you can confirm with:<br />
<br />
<pre><br />
[root@newbackup3 ~]# cli64 vsf info<br />
# Name Raid Name Level Capacity Ch/Id/Lun State<br />
===============================================================================<br />
1 ARC-1160-VOL#00 Raid Set # 00 Raid5 5000.0GB 00/00/00 Checking(22.5%)<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
[root@newbackup3 ~]#<br />
</pre><br />
<br />
== Update OS list ==<br />
<br />
# check for any new VZ templates we want to offer: <tt>vzup2date -z</tt><br />
# see if there's any OS's we want to include in our colo install list. Update 2 places: <tt>signup/html/colo_quote.html</tt> & <tt>signup/html/step1.html</tt><br />
# update the mgmt database (ref_templates table, ref_systems table).<br />
<br />
= Infrequent tasks =<br />
<br />
== Free up space on gateway ==<br />
<br />
<pre>newgateway /var/spool# cd clientmqueue/<br />
newgateway /var/spool/clientmqueue# sh<br />
# for f in `ls`; do rm $f; done<br />
exit</pre><br />
<br />
== Free up space on mail ==<br />
<br />
You can clear out root mail:<br />
<pre><br />
mail /var/log# ll -h /var/mail/root<br />
-rw------- 1 root mail 543K Dec 19 13:05 /var/mail/root<br />
mail /var/log# rm /var/mail/root<br />
</pre><br />
<br />
Or you can archive mail logs:<br />
<pre><br />
mail /var/log# ls -l htt*<br />
-rw-r--r-- 1 root wheel 297436931 Dec 19 13:26 httpd-access.log<br />
-rw-r--r-- 1 root wheel 9824324 Jul 4 11:34 httpd-access.log.old.0.gz<br />
-rw-r--r-- 1 root wheel 6884137 Mar 17 2012 httpd-access.log.old.1.gz<br />
-rw-r--r-- 1 root wheel 18557444 Dec 3 2009 httpd-access.log.old.10.gz<br />
-rw-r--r-- 1 root wheel 14740263 Jan 9 2007 httpd-access.log.old.11.gz<br />
-rw-r--r-- 1 root wheel 14209465 Nov 28 2007 httpd-access.log.old.12.gz<br />
-rw-r--r-- 1 root wheel 16874396 Feb 19 2012 httpd-access.log.old.3.gz<br />
-rw-r--r-- 1 root wheel 14554859 Jul 22 2011 httpd-access.log.old.4.gz<br />
-rw-r--r-- 1 root wheel 10513227 Feb 18 2011 httpd-access.log.old.5.gz<br />
-rw-r--r-- 1 root wheel 7201946 Oct 29 2010 httpd-access.log.old.6.gz<br />
-rw-r--r-- 1 root wheel 10062537 May 6 2010 httpd-access.log.old.7.gz<br />
-rw-r--r-- 1 root wheel 10157042 Aug 12 2010 httpd-access.log.old.8.gz<br />
-rw-r--r-- 1 root wheel 11909534 Mar 4 2010 httpd-access.log.old.9.gz<br />
-rw-r--r-- 1 root wheel 59030930 Dec 19 13:01 httpd-error.log<br />
-rw-r--r-- 1 root wheel 3413134 Mar 4 2010 httpd-error.log.0.gz<br />
-rw-r--r-- 1 root wheel 795515 May 1 2007 httpd-error.log.1.gz<br />
-rw-r--r-- 1 root wheel 1142153 Nov 30 2007 httpd-error.log.2.gz<br />
-rw-r--r-- 1 root wheel 2325801 Feb 18 2011 httpd-error.log.gz<br />
<br />
mail /var/log# sh<br />
# for f in 12 11 10 9 8 7 6 5 4 3 2 1 0; do g=`echo $f+1|bc`; mv httpd-access.log.old.$f.gz httpd-access.log.old.$g.gz; done<br />
# mv httpd-access.log httpd-access.log.old.0<br />
# touch httpd-access.log<br />
# apachectl restart<br />
# gzip httpd-access.log.old.0<br />
<br />
# for f in 2 1 0; do g=`echo $f+1|bc`; mv httpd-error.log.$f.gz httpd-error.log.$g.gz; done<br />
# mv httpd-error.log httpd-error.log.0<br />
# touch httpd-error.log<br />
# apachectl restart<br />
# gzip httpd-error.log.0<br />
# exit<br />
</pre><br />
<br />
== Free up space on bwdb2 ==<br />
<br />
You can either remove items from <tt>/usr/home/archive</tt> or you can scp them to backup3:/data/bwdb2/archive .<br />
<br />
== Free up space on backup1 ==<br />
<br />
backup1 is our primary customer backup system. As usage grows over time, it needs to be regularly purged of old files. The easiest way to do this is by removing deprecated files. These mostly consist of cancelled customers or temporary dump/storage files (created during dump/restores). Our standard policy is to hang onto cancelled customers for 6mos after which we remove their files (as far as customers know their data is purged immediately, but we hang onto it just in case.. and in some cases we cancel a server due to non payment so this makes it easy to restore their system). To find files to remove:<br />
<br />
<pre>[root@backup1 ~]# cd /data/deprecated/<br />
[root@backup1 /data/deprecated]# ls<br />
2101-migrated-20120317.tgz old-683-cxld-20121021.tgz<br />
69.55.230.2-wwwbackup old-744-cxld-20120708.tgz<br />
991-DONTDELETE.tgz old-809-cxld-20120609.tgz<br />
archive-col02050-mdfile-cxld-20120409.gz old-854-cxld-20120621.tgz<br />
col01371.tgz old-931-cxld-20060513.tgz<br />
deleteme_ubuntu-10.10-x86_20111205 old-col00123-mdfile-noarchive-20120417.gz<br />
jail10_old old-col00147-vnfile-cxld-20120828.gz<br />
jail14_rsync_old old-col00419-dump-cxld-20120224.gz<br />
jail15_old old-col01098-vnfile-cxld-20120827.gz<br />
jail3_old old-col01278-dump-cxld-20120822<br />
jail4_old old-col01517-dump-cxld-20120828<br />
jail5_old old-col01669-dump-cxld-20120203.gz<br />
old-1009-cxld-20120608.tgz old-col01687-dump-cxld-20120909<br />
old-1012-cxld-20120411.tgz old-col01790-dump-cxld-20120828<br />
old-1052-cxld-20120721.tgz old-col01812-dump-cxld-20120820<br />
old-10631-cxld-20120622.tgz old-col01938-mdfile-cxld-20120619.gz<br />
old-10632-cxld-20120622.tgz old-col02095-mdfile-noarchive-20120523.gz<br />
old-10633-cxld-20120622.tgz olddebian-3.0-v15-20110610.tgz<br />
old-1236-cxld-20120621.tgz oldmod_frontpage-deb30-v15-20110610.tgz<br />
old-1381-cxld-20120404.tgz oldmod_perl-deb30-v15-20110610.tgz<br />
old-1422-cxld-20120721.tgz oldmod_ssl-deb30-v15-20110610.tgz<br />
old-14681-cxld-20120619.tgz oldmysql-deb30-v15-20110610.tgz<br />
old-1544-cxld-20120626.tgz oldproftpd-deb30-v15-20110610.tgz<br />
old-18351-cxld-20120605.tgz old_virt14<br />
old-1853-cxld-20120910.tgz old_virt18<br />
old-1963-cxld-20120206.tgz oldwebmin-deb30-v15-20110610.tgz<br />
old-1967-cxld-20120605.tgz suse.virt11.20120421.tgz<br />
old-1981-noarchive-20120729.tgz virt11<br />
old-2030-migrated-noarchive-20120727.tgz virt12_old<br />
old-2037-cxld-20120716.tgz virt13_old<br />
old-2065-cxld-20120727.tgz virt16_old<br />
old-2068-cxld-20120424.tgz virt4_old<br />
old-2085-cxld-20120531.tgz virt5_old<br />
old-364-cxld-20120904.tgz virt6_old<br />
old-446-cxld-20120512.tgz virt7_old<br />
old-613-cxld-20120601.tgz virt8_old<br />
[root@backup1 /data/deprecated]#<br />
</pre><br />
<br />
virtX_old and jailX_old are permanently archived, so ignore those as well as anything else marked not to delete or otherwise suspicious. Likewise, probably a good idea to try to hang onto oldTEMPLATE.gz as long as we can as well.<br />
Most of the stuff we want to delete is dated when it was deprecated, making this easy. So to remove files from 6 mos ago (running this in Oct):<br />
<br />
<pre>[root@backup1 /data/deprecated]# ls old*201204*<br />
old-1012-cxld-20120411.tgz old-2068-cxld-20120424.tgz<br />
old-1381-cxld-20120404.tgz old-col00123-mdfile-noarchive-20120417.gz<br />
[root@backup1 /data/deprecated]# rm old*201204*<br />
</pre><br />
<br />
Every few months you will also want to remove some of the snapshot archives for mail. We typically save the 1st, 10th, and 20th of each month. To do this you set aside the dates you want to save then remove months at a time, followed by restoring the set aside dates. Here's how that works:<br />
<pre>[root@backup1 /data/www/daily]# ls<br />
05 08-10-11 10-04-10 11-10-10 12-07-29 12-09-21 12-11-14<br />
06 08-10-21 10-04-20 11-10-20 12-07-30 12-09-22 12-11-15<br />
06-06-01-usr-home.tgz 08-11-01 10-05-01 11-11-01 12-07-31 12-09-23 12-11-16<br />
06-07-01-usr-home.tgz 08-11-10 10-05-11 11-11-10 12-08-01 12-09-24 12-11-17<br />
06-08-01-usr-home.tgz 08-11-20 10-05-20 11-11-20 12-08-02 12-09-25 12-11-18<br />
06-09-01-usr-home.tgz 08-12-01 10-06-01 11-12-01 12-08-03 12-09-26 12-11-19<br />
06-11-10 08-12-10 10-06-10 11-12-10 12-08-04 12-09-27 12-11-20<br />
06-12-21 08-12-20 10-06-20 11-12-20 12-08-05 12-09-28 12-11-21<br />
07-01-10 09-01-01 10-07-01 12-01-01 12-08-06 12-09-29 12-11-22<br />
07-01-20 09-01-10 10-07-10 12-01-10 12-08-07 12-09-30 12-11-23<br />
07-02-10 09-01-20 10-07-20 12-01-20 12-08-08 12-10-01 12-11-24<br />
07-02-20 09-02-01 10-08-01 12-02-01 12-08-09 12-10-02 12-11-25<br />
07-03-01 09-02-10 10-08-10 12-02-10 12-08-10 12-10-03 12-11-26<br />
07-03-20 09-02-20 10-08-20 12-02-20 12-08-11 12-10-04 12-11-27<br />
07-04-01 09-03-01 10-09-01 12-03-01 12-08-12 12-10-05 12-11-28<br />
07-04-10 09-03-10 10-09-10 12-03-10 12-08-13 12-10-06 12-11-29<br />
07-04-20 09-03-20 10-09-20 12-03-20 12-08-14 12-10-07 12-11-30<br />
07-05-01 09-04-01 10-10-01 12-04-01 12-08-15 12-10-08 12-12-01<br />
07-05-10 09-04-10 10-10-10 12-04-10 12-08-16 12-10-09 12-12-02<br />
07-05-20 09-04-20 10-10-20 12-04-20 12-08-17 12-10-10 12-12-03<br />
07-06-01 09-05-01 10-11-01 12-05-01 12-08-18 12-10-11 12-12-04<br />
07-06-10 09-05-10 10-11-10 12-05-10 12-08-19 12-10-12 12-12-05<br />
07-06-20 09-05-20 10-11-20 12-05-20 12-08-20 12-10-13 12-12-06<br />
07-07-20 09-06-01 10-12-01 12-06-01 12-08-21 12-10-14 12-12-07<br />
07-08-10 09-06-10 10-12-10 12-06-10 12-08-22 12-10-15 12-12-08<br />
07-08-20 09-06-20 10-12-20 12-06-20 12-08-23 12-10-16 12-12-09<br />
07-09-01 09-07-01 11-01-01 12-07-01 12-08-24 12-10-17 12-12-10<br />
07-10-01 09-07-10 11-01-10 12-07-02 12-08-25 12-10-18 12-12-11<br />
07-10-10 09-07-20 11-01-21 12-07-03 12-08-26 12-10-19 12-12-12<br />
07-10-20 09-08-01 11-02-01 12-07-04 12-08-27 12-10-20 12-12-13<br />
07-12-01 09-08-10 11-02-10 12-07-05 12-08-28 12-10-21 12-12-14<br />
07-12-10 09-08-20 11-02-20 12-07-06 12-08-29 12-10-22 12-12-15<br />
08-01-01 09-09-01 11-03-01 12-07-07 12-08-30 12-10-23 12-12-16<br />
08-01-20 09-09-10 11-03-10 12-07-08 12-08-31 12-10-24 12-12-17<br />
08-02-20 09-09-20 11-03-20 12-07-09 12-09-01 12-10-25 12-12-18<br />
08-03-01 09-10-01 11-04-01 12-07-10 12-09-02 12-10-26 12-12-19<br />
08-03-10 09-10-10 11-04-10 12-07-11 12-09-03 12-10-27 12-12-20<br />
08-03-20 09-10-20 11-04-20 12-07-12 12-09-04 12-10-28 12-12-21<br />
08-04-01 09-11-01 11-05-01 12-07-13 12-09-05 12-10-29 12-12-22<br />
08-04-20 09-11-10 11-05-10 12-07-14 12-09-06 12-10-30 12-12-23<br />
08-05-01 09-11-20 11-05-20 12-07-15 12-09-07 12-10-31 12-12-24<br />
08-05-10 09-12-01 11-06-01 12-07-16 12-09-08 12-11-01 12-12-25<br />
08-06-10 09-12-10 11-06-10 12-07-17 12-09-09 12-11-02 12-12-26<br />
08-06-20 09-12-20 11-06-20 12-07-18 12-09-10 12-11-03 12-12-27<br />
08-07-02 10-01-01 11-07-01 12-07-19 12-09-11 12-11-04 12-12-28<br />
08-07-10 10-01-10 11-07-10 12-07-20 12-09-12 12-11-05 2008-10-23<br />
08-07-20 10-01-20 11-07-20 12-07-21 12-09-13 12-11-06 bb.tgz<br />
08-08-01 10-02-01 11-08-01 12-07-22 12-09-14 12-11-07 boot<br />
08-08-10 10-02-10 11-08-10 12-07-23 12-09-15 12-11-08 current<br />
08-08-21 10-02-20 11-08-20 12-07-24 12-09-16 12-11-09 hold<br />
08-09-01 10-03-01 11-09-01 12-07-25 12-09-17 12-11-10<br />
08-09-10 10-03-10 11-09-10 12-07-26 12-09-18 12-11-11<br />
08-09-21 10-03-20 11-09-20 12-07-27 12-09-19 12-11-12<br />
08-10-01 10-04-01 11-10-01 12-07-28 12-09-20 12-11-13<br />
[root@backup1 /data/www/daily]#<br />
</pre><br />
<br />
So we see that everything up to July 2012 has been pruned. To prune July 2012 we do the following:<br />
<br />
<pre>mv 12-07-01 hold<br />
mv 12-07-10 hold<br />
mv 12-07-20 hold<br />
rm -fr 12-07*<br />
mv hold/* .</pre></div>75.80.128.147https://wiki.jcihosting.com/index.php?title=Routine_Maintenance&diff=2200Routine Maintenance2020-02-25T17:41:33Z<p>75.80.128.147: /* check bb for warnings */</p>
<hr />
<div>= Daily Tasks =<br />
<br />
== check load graphs == <br />
<br />
Click on the Load link in mgmt <br />
<br />
This screen shows you load levels on our servers and network traffic for critical machines (firewalls, backup servers).<br />
<br />
If you see load high or increasing <br />
<br />
FreeBSD: <br />
run [[VPS_Management#jtop|jtop]] (or [[VPS_Management#jt|jt]] > 7.x) and see if there are any runaway processes. Here are some examples of entries in top that are<br />
definitely runaway processes:<br />
<br />
<pre>79481 root 64 0 2256K 1056K CPU1 1 58:16 87.40% 87.40% nano<br />
50650 1000 64 0 1852K 1112K RUN 0 207.9H 84.08% 84.08% screen<br />
14829 www 2 0 39100K 31736K accept 0 104:24 46.54% 6.54% httpd<br />
42065 root 61 0 1300K 844K RUN 1 47.8H 91.36% 91.36% ee<br />
1328 www 56 0 18440K 10796K CPU1 0 64.4H 97.71% 97.71% httpd<br />
26251 user 57 0 6124K 1160K CPU1 1 82.9H 98.44% 98.44% screen<br />
89874 root 60 0 1352K 892K RUN 1 33.8H 65.82% 65.82% dialog<br />
38656 1000 64 0 3088K 2136K CPU0 0 806:13 97.95% 97.95% StutBot<br />
27630 root 64 0 1396K 972K RUN 1 76.8H 86.47% 86.47% ee</pre><br />
<br />
Linux:<br />
run [[VPS_Management#vwe|vwe]] to see which VPS’s have high loads. From there run <tt>[[VPS_Management#vp|vp]] <veid></tt> and/or <tt>[[VPS_Management#vt|vt]] <veid></tt> to see what's going on in that system.<br />
[[VPS_Management#vzstat|vzstat]] will also give you a nice picture of whats going on, systems with high numbers in the mlat column are likely culprits.<br />
<br />
examples of out of control procs:<br />
<br />
<pre>12183 nobody 16 0 4916 1348 1340 R 45.5 0.0 4249m httpd<br />
29266 #502 16 0 1852 796 792 R 22.5 0.0 1104m vim<br />
23860 #41 16 0 5472 5472 2076 R 98.9 0.2 31:41 python<br />
19227 bin 19 0 1688 716 652 R 99.9 0.0 321:08 wtrs_ui<br />
7762 apache 16 0 268 236 224 R 85.7 0.0 1010m ptrace<br />
4624 #501 20 0 4304 2400 2044 R 53.6 0.1 284:32 YoSucker<br />
20451 #506 20 0 1876 820 816 R 17.2 0.0 169:35 vim<br />
8834 #514 20 0 900 724 672 R 77.6 0.0 382:30 neostats<br />
31815 apache 14 0 3176 3176 1696 R 74.4 0.1 6:15 counter</pre><br />
<br />
Just kill -9 them and be done with it.<br />
<br />
Also, anytime you see `kmod` or `ptrace` - kill those immediaely no matter how much they are using - they are attempts to exploit the linux ptrace bug. They won't work, but they suck a lot of CPU...<br />
<br />
Also, any other processes that are at 90-100% cpu usage and have been running for any long period of time should be killed except for mysqld processes on FreeBSD. See above.<br />
<br />
However, there is an exception:<br />
<br />
if it is a mysqld, we don't want to kill their database. What you want to do is <tt>[[VPS_Management#jpid|jpid]] <pid></tt> to see who owns it, and then email them the paste containing the instructions for the nanny. Or you can simply do a <tt>kill -1 PID</tt> on the process to restart it.<br />
<br />
=== Load averages jump at night ===<br />
<br />
The load averages on the FreeBSD systems may jump up at night between 1 and 4 am - this is because the backups are running - if this is what is causing a jump in load, you will see processes like `rsync` in top eating a lot of CPU time.<br />
<br />
== check backups ==<br />
<br />
mgmt -> Motnroing -> Backups and make sure every machine was backed up the previous nite. <br />
Also look at df on backup1 and backup2 to make sure no disk is approaching full, though bb should warn us in advance. Please note - errors encountered when a backup script on any of the particular systems run will gnerate an email to support@johncompanies.com so you can know immediately the day after if the directory to be backed up has been moved or no longer exists. A paste exists for this to notify the customer of a non-existant file/dir.<br />
<br />
== check bb for warnings ==<br />
mgmt -> BigBrother<br />
<br />
Some events don't generate pages (on purpose). You will only see them by going to the bb main page.<br />
<br />
== check jail5 for crashed VPSs ==<br />
<br />
On jail5<br />
<br />
notrunning<br />
<br />
To restart a VPS <br />
<br />
vm restart col0XXXX<br />
<br />
= Monthly Tasks =<br />
== rotate pine sent mail (1st of month) ==<br />
On the 1st of the month, before any emails are sent out, quit out of pine, then log back in. Send mail from last month will be archived.<br />
If you mess up and do it on the 3rd (for example), you can go into the previous month's saved email and save emails from the current month into the <tt>sent-mail</tt> (current month) mailbox.<br />
<br />
== b/w caps ==<br />
On the 1st: remove any bwcaps put into the firewall (only really applies if a bwcap was added cause someone went over on b/w):<br />
<br />
ipfw list|grep pipe<br />
ipfw del [each rule listed]<br />
<br />
NOTE: this cronjob on newgateway will do some of that for you, provided you used one of the following pipe #s:<br />
0 0 1 * * /sbin/ipfw del 3 4 5 17331<br />
<br />
-------------------<br />
<br />
We really don’t do this anymore since we have centralized traffic accounting with netflow, but for posterity:<br />
<br />
Make sure all machines reset counters to 0 after midnight on the 1st<br />
Make sure they dumped a counter<br />
<br />
On each jail run:<br />
trafficgather.pl<br />
<br />
And on each virt:<br />
linuxtrafficgather.pl<br />
<br />
== Monthly RAID checks ==<br />
<br />
Every month we check the health of and verfy the parity on all our RAID-based systems.<br />
To facilitate this, we've created a simple script to start the process:<br />
<br />
sh /root/verify.sh<br />
<br />
=== Adaptec controllers ===<br />
Here's some sample output:<br />
<pre><br />
mail /usr/local/www/scripts# sh /root/verify.sh<br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> container list /f<br />
Executing: container list /full=TRUE<br />
Num Total Oth Chunk Scsi Partition<br />
Creation System<br />
Label Type Size Ctr Size Usage B:ID:L Offset:Size State RO Lk Task Done% Ent<br />
Date Time Files<br />
----- ------ ------ --- ------ ------- ------ ------------- ------- -- -- ------- ------ ---<br />
------ -------- ------<br />
0 Mirror 33.9GB Open 0:01:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:32<br />
/dev/aacd0 mirror0 0:00:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:32<br />
<br />
1 Mirror 33.9GB Open 0:02:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:50<br />
/dev/aacd1 mirror1 0:03:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:50<br />
<br />
<br />
AAC0> disk list /f<br />
Executing: disk list /full=TRUE<br />
<br />
B:ID:L Device Type Removable media Vendor-ID Product-ID Rev Blocks Bytes/Bl<br />
ock Usage Shared Rate<br />
------ -------------- --------------- --------- ---------------- ----- --------- --------<br />
--- ---------------- ------ ----<br />
0:00:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:01:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:02:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
0:03:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512<br />
Initialized NO 160<br />
<br />
AAC0> disk show smart<br />
Executing: disk show smart<br />
<br />
Smart Method of Enable<br />
Capable Informational Exception Performance Error<br />
B:ID:L Device Exceptions(MRIE) Control Enabled Count<br />
------ ------- ---------------- --------- ----------- ------<br />
0:00:0 Y 6 Y N 0<br />
0:01:0 Y 6 Y N 0<br />
0:02:0 Y 6 Y N 0<br />
0:03:0 Y 6 Y N 0<br />
0:06:0 N<br />
<br />
AAC0> task list<br />
Executing: task list<br />
<br />
Controller Tasks<br />
<br />
TaskId Function Done% Container State Specific1 Specific2<br />
------ -------- ------- --------- ----- --------- ---------<br />
<br />
No tasks currently running on controller<br />
<br />
AAC0> dia sh hi<br />
Executing: diagnostic show history<br />
No switches specified, defaulting to "/current".<br />
<br />
<br />
<br />
*** HISTORY BUFFER FROM CURRENT CONTROLLER RUN ***<br />
<br />
[00]: GetDiskLogEntry: container - 1, entry return 0<br />
[01]: Container 1 started SCRUB task<br />
[02]: Starting Mirror:1 scrub<br />
[03]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[04]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[05]: UpdateDiskLogIndex - Set - container 0, index 1<br />
[06]: GetDiskLogEntry: container - 0, entry return 1<br />
[07]: Container 0 started SCRUB task<br />
[08]: Starting Mirror:0 scrub<br />
[09]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[10]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[11]: Mirror Scrub Container:1 ErrorsFound:0<br />
[12]: Clear disk log: sector - 80, driveno 2<br />
[13]: Clear disk log: sector - 80, driveno 3<br />
[14]: Container 1 completed SCRUB task:<br />
[15]: Mirror Scrub Container:0 ErrorsFound:0<br />
[16]: Clear disk log: sector - 81, driveno 1<br />
[17]: Clear disk log: sector - 81, driveno 0<br />
[18]: Container 0 completed SCRUB task:<br />
[19]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[20]: GetDiskLogEntry: container - 0, entry return 0<br />
[21]: Container 0 started SCRUB task<br />
[22]: Starting Mirror:0 scrub<br />
[23]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[24]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[25]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[26]: GetDiskLogEntry: container - 1, entry return 1<br />
[27]: Container 1 started SCRUB task<br />
[28]: Starting Mirror:1 scrub<br />
[29]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[30]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[31]: Mirror Scrub Container:1 ErrorsFound:0<br />
[32]: Clear disk log: sector - 81, driveno 2<br />
[33]: Clear disk log: sector - 81, driveno 3<br />
[34]: Container 1 completed SCRUB task:<br />
[35]: Mirror Scrub Container:0 ErrorsFound:0<br />
[36]: Clear disk log: sector - 80, driveno 1<br />
[37]: Clear disk log: sector - 80, driveno 0<br />
[38]: Container 0 completed SCRUB task:<br />
[39]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[40]: GetDiskLogEntry: container - 0, entry return 0<br />
[41]: Container 0 started SCRUB task<br />
[42]: Starting Mirror:0 scrub<br />
[43]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[44]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[45]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[46]: GetDiskLogEntry: container - 1, entry return 1<br />
[47]: Container 1 started SCRUB task<br />
[48]: Starting Mirror:1 scrub<br />
[49]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[50]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[51]: Mirror Scrub Container:1 ErrorsFound:0<br />
[52]: Clear disk log: sector - 81, driveno 2<br />
[53]: Clear disk log: sector - 81, driveno 3<br />
[54]: Container 1 completed SCRUB task:<br />
[55]: Mirror Scrub Container:0 ErrorsFound:0<br />
[56]: Clear disk log: sector - 80, driveno 1<br />
[57]: Clear disk log: sector - 80, driveno 0<br />
[58]: Container 0 completed SCRUB task:<br />
[59]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[60]: GetDiskLogEntry: container - 0, entry return 0<br />
[61]: Container 0 started SCRUB task<br />
[62]: Starting Mirror:0 scrub<br />
[63]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[64]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[65]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[66]: GetDiskLogEntry: container - 1, entry return 1<br />
[67]: Container 1 started SCRUB task<br />
[68]: Starting Mirror:1 scrub<br />
[69]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[70]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[71]: Mirror Scrub Container:1 ErrorsFound:0<br />
[72]: Clear disk log: sector - 81, driveno 2<br />
[73]: Clear disk log: sector - 81, driveno 3<br />
[74]: Container 1 completed SCRUB task:<br />
[75]: Mirror Scrub Container:0 ErrorsFound:0<br />
[76]: Clear disk log: sector - 80, driveno 1<br />
[77]: Clear disk log: sector - 80, driveno 0<br />
[78]: Container 0 completed SCRUB task:<br />
[79]: UpdateDiskLogIndex - Set - container 0, index 0<br />
[80]: GetDiskLogEntry: container - 0, entry return 0<br />
[81]: Container 0 started SCRUB task<br />
[82]: Starting Mirror:0 scrub<br />
[83]: Master disk: 1, start sector: 128, sector count = 71286784<br />
[84]: Slave disk: 0, start sector: 128, sector count = 71286784<br />
[85]: UpdateDiskLogIndex - Set - container 1, index 1<br />
[86]: GetDiskLogEntry: container - 1, entry return 1<br />
[87]: Container 1 started SCRUB task<br />
[88]: Starting Mirror:1 scrub<br />
[89]: Master disk: 2, start sector: 128, sector count = 71286784<br />
[90]: Slave disk: 3, start sector: 128, sector count = 71286784<br />
[91]: Mirror Scrub Container:1 ErrorsFound:0<br />
[92]: Clear disk log: sector - 81, driveno 2<br />
[93]: Clear disk log: sector - 81, driveno 3<br />
[94]: Container 1 completed SCRUB task:<br />
[95]: Mirror Scrub Container:0 ErrorsFound:0<br />
[96]: Clear disk log: sector - 80, driveno 1<br />
[97]: Clear disk log: sector - 80, driveno 0<br />
[98]: Container 0 completed SCRUB task:<br />
[99]:<br />
<br />
========================<br />
History Output Complete.<br />
<br />
AAC0><br />
AAC0> exit<br />
Executing: exit<br />
<br />
press enter when ready to run verify <INS><br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> contai scr 0<br />
Executing: container scrub 0<br />
<br />
AAC0> contai scr 1<br />
Executing: container scrub 1<br />
<br />
AAC0> exit<br />
Executing: exit<br />
<br />
when done run: <br />
<br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
<br />
<br />
Nov 1 10:32:46 mail /kernel: aac0: **Monitor** Container 0 started SCRUB task<br />
Nov 1 10:32:47 mail /kernel: aac0: **Monitor** Container 1 started SCRUB task<br />
<br />
</pre><br />
<br />
Here's an analysis of what we're seeing and what we're looking for:<br />
<br />
<pre><br />
AAC0> container list /f<br />
Executing: container list /full=TRUE<br />
Num Total Oth Chunk Scsi Partition<br />
Creation System<br />
Label Type Size Ctr Size Usage B:ID:L Offset:Size State RO Lk Task Done% Ent<br />
Date Time Files<br />
----- ------ ------ --- ------ ------- ------ ------------- ------- -- -- ------- ------ ---<br />
------ -------- ------<br />
0 Mirror 33.9GB Open 0:01:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:32<br />
/dev/aacd0 mirror0 0:00:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:32<br />
<br />
1 Mirror 33.9GB Open 0:02:0 64.0KB:33.9GB Normal 0<br />
071002 05:39:50<br />
/dev/aacd1 mirror1 0:03:0 64.0KB:33.9GB Normal 1<br />
071002 05:39:50<br />
</pre><br />
<br />
This is showing you the health of the arrays. You're looking for ''Normal'' under the State column, and the absence of a ! in the sector size - sometimes, you'll see this:<br />
64.0KB!33.9GB <br />
<br />
That indicates a problem.<br />
<br />
<pre><br />
AAC0> disk show smart<br />
Executing: disk show smart<br />
<br />
Smart Method of Enable<br />
Capable Informational Exception Performance Error<br />
B:ID:L Device Exceptions(MRIE) Control Enabled Count<br />
------ ------- ---------------- --------- ----------- ------<br />
0:00:0 Y 6 Y N 0<br />
0:01:0 Y 6 Y N 0<br />
0:02:0 Y 6 Y N 0<br />
0:03:0 Y 6 Y N 0<br />
0:06:0 N<br />
</pre><br />
<br />
This shows you a SMART report output. Looking for values in the Error Count column.<br />
<pre><br />
AAC0> task list<br />
Executing: task list<br />
<br />
Controller Tasks<br />
<br />
TaskId Function Done% Container State Specific1 Specific2<br />
------ -------- ------- --------- ----- --------- ---------<br />
<br />
No tasks currently running on controller<br />
</pre><br />
<br />
Look for absence of tasks running- a bad thing would be to see a rebuild or verify running when you didn't initiate it.<br />
<br />
With the history output, you're looking for any anomalies or events since the last time a verify was run. If you see a drive with lots of problems, you may want to take backups before allowing the verify to run since it could replicate errors onto the good drive.<br />
<br />
After you see the history output, it will prompt you to press enter to run the verify. If you're happy with all the output you're seeing- mirror is healthy, history looks good, it's safe to proceed. Otherwise ^C to exit. After hitting enter it will start the verify and start to tail the messages log file (so you can easily see when the verify is complete). Here's what that'll look like:<br />
<br />
<pre><br />
Nov 1 14:38:08 mail /kernel: aac0: **Monitor** Container 1 completed SCRUB task:<br />
Nov 1 14:46:45 mail /kernel: aac0: **Monitor** Container 0 completed SCRUB task:<br />
</pre><br />
<br />
So, putting it all together, after hitting enter to start the verify, you'll see:<br />
<br />
<pre><br />
---------------------------------------------------------------------------------------------<br />
<br />
Adaptec SCSI RAID Controller Command Line Interface<br />
Copyright 1998-2002 Adaptec, Inc. All rights reserved<br />
---------------------------------------------------------------------------------------------<br />
<br />
<br />
CLI > open aac0<br />
Executing: open "aac0"<br />
<br />
AAC0> contai scr 0<br />
Executing: container scrub 0<br />
<br />
AAC0> contai scr 1<br />
Executing: container scrub 1<br />
<br />
AAC0> exit<br />
Executing: exit<br />
<br />
when done run: <br />
<br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
<br />
<br />
Nov 1 10:32:46 mail /kernel: aac0: **Monitor** Container 0 started SCRUB task<br />
Nov 1 10:32:47 mail /kernel: aac0: **Monitor** Container 1 started SCRUB task<br />
<br />
</pre><br />
<br />
When the scrub(s) (verify) are complete - if the server has multiple logical drives, it will run both in parallel - you should exit the tail of the log file (^C) and run:<br />
<br />
<pre><br />
aaccli<br />
open aac0<br />
dia sh hi<br />
c<br />
</pre><br />
<br />
Which will show you the diagnostic history, you're looking for the results of the most recent scrub:<br />
<br />
<pre><br />
[100]: Mirror Scrub Container:1 ErrorsFound:0<br />
[101]: Clear disk log: sector - 81, driveno 2<br />
[102]: Clear disk log: sector - 81, driveno 3<br />
[103]: Container 1 completed SCRUB task:<br />
[104]: Mirror Scrub Container:0 ErrorsFound:0<br />
[105]: Clear disk log: sector - 80, driveno 1<br />
[106]: Clear disk log: sector - 80, driveno 0<br />
[107]: Container 0 completed SCRUB task:<br />
</pre><br />
<br />
^C to exit the RAID CLI.<br />
<br />
If you see:<br />
<br />
[104]: Mirror Scrub Container:0 ErrorsFound:5<br />
<br />
You'll want to rerun the verify on that drive till it shows 0, or perhaps replace the drive- you should be able to see from the output which drive had the problem.<br />
<br />
Depending on the size and how busy the drive is, the verify can take anywhere from an hour to the better part of a day.<br />
<br />
You will notice that the diagnostic history is not shown on our modern adaptec cards (i.e. any adaptec card not in a Dell 2450). The reason for this is the history is never cleared, so there's simply too much data to show and it just crashes the CLI. So, don't bother trying to see it...which does make it hard to see if there are problems going on, so you just need to watch the scrub to see it goes to 100%. You will also notice that on some servers there's no tail of messages. Again, this is cause no data is shown there about the completion of the scrub. The thing to do here is to go into the CLI and continue to <tt>show tasks</tt> to monitor scrub progress.<br />
<br />
See [[RAIC_CLI#Adaptec|Adaptec RAID CLI Reference]] for more details on how to use the CLI<br />
<br />
=== DELL (LSI-based) SAS controllers ===<br />
<br />
Here's what the output looks like when running verify.sh on a LSI-based card:<br />
<br />
<pre><br />
jail2 /mnt/data2# sh /root/verify.sh<br />
<br />
Adapter #0<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 0<br />
Device Id: 0<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018396142<br />
SAS Address(1): 0x0<br />
Connected Port Number: 0(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DESN<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 1<br />
Device Id: 1<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018395db2<br />
SAS Address(1): 0x0<br />
Connected Port Number: 1(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DERV<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 2<br />
Device Id: 2<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50006eece89<br />
SAS Address(1): 0x0<br />
Connected Port Number: 2(path0)<br />
Inquiry Data: SEAGATE ST3300555SS T2113LM4BFBZ<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 3<br />
Device Id: 3<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50006eee035<br />
SAS Address(1): 0x0<br />
Connected Port Number: 3(path0)<br />
Inquiry Data: SEAGATE ST3300555SS T2113LM4BGF7<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 4<br />
Device Id: 4<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x5000c50004bd7ea5<br />
SAS Address(1): 0x0<br />
Connected Port Number: 4(path0)<br />
Inquiry Data: SEAGATE ST3300656SS HS093QP0G8SW<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
Enclosure Device ID: 32<br />
Slot Number: 5<br />
Device Id: 5<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 286102MB [0x22ecb25c Sectors]<br />
Non Coerced Size: 285590MB [0x22dcb25c Sectors]<br />
Coerced Size: 285568MB [0x22dc0000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e01f1c4112<br />
SAS Address(1): 0x0<br />
Connected Port Number: 5(path0)<br />
Inquiry Data: FUJITSU MBA3300RC D306BJ15P9201W06<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 0 (Target Id: 0)<br />
Name:<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:139392MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 1 (Target Id: 1)<br />
Name:MIRROR1<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
<br />
<br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 2 (Target Id: 2)<br />
Name:MIRROR2<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
<br />
Exit Code: 0x00<br />
Battery FRU : N/A<br />
Battery Warning : Enabled<br />
Memory Correctable Errors : 0<br />
Memory Uncorrectable Errors : 0<br />
BBU : Present<br />
BBU : Yes<br />
Cache When BBU Bad : Disabled<br />
press enter when ready to run verify<br />
<br />
</pre><br />
<br />
Before pressing enter, here's what we're looking for:<br />
<br />
<pre><br />
Enclosure Device ID: 32<br />
Slot Number: 0<br />
Device Id: 0<br />
Sequence Number: 2<br />
Media Error Count: 0<br />
Other Error Count: 0<br />
Predictive Failure Count: 0<br />
Last Predictive Failure Event Seq Number: 0<br />
PD Type: SAS<br />
Raw Size: 140014MB [0x11177328 Sectors]<br />
Non Coerced Size: 139502MB [0x11077328 Sectors]<br />
Coerced Size: 139392MB [0x11040000 Sectors]<br />
Firmware state: Online<br />
SAS Address(0): 0x500000e018396142<br />
SAS Address(1): 0x0<br />
Connected Port Number: 0(path0)<br />
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DESN<br />
Foreign State: None<br />
Media Type: Hard Disk Device<br />
Device Speed: Unknown<br />
Link Speed: Unknown<br />
</pre><br />
<br />
This is the output shown for each physical drive in the system. We're looking to confirm it's Firmware state is ''Online'', and Media Error Count, Other Error Count, and Predictive Failure Count are all zero (or near zero).<br />
<br />
<pre><br />
Adapter 0 -- Virtual Drive Information:<br />
Virtual Disk: 1 (Target Id: 1)<br />
Name:MIRROR1<br />
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0<br />
Size:285568MB<br />
State: Optimal<br />
Stripe Size: 64kB<br />
Number Of Drives:2<br />
Span Depth:1<br />
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU<br />
Access Policy: Read/Write<br />
Disk Cache Policy: Disk's Default<br />
</pre><br />
<br />
This is the output for each logical drive. We're looking for State ''Optimal''. Also confirm <tt>Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU</tt><br />
<br />
<pre><br />
Exit Code: 0x00<br />
Battery FRU : N/A<br />
Battery Warning : Enabled<br />
Memory Correctable Errors : 0<br />
Memory Uncorrectable Errors : 0<br />
BBU : Present<br />
BBU : Yes<br />
Cache When BBU Bad : Disabled<br />
</pre><br />
<br />
Confirm that the battery is present and error-free.<br />
<br />
If all that checks out, you're ready to proceed with the verify. After pressing enter, the verify is started and here's what you see:<br />
<pre><br />
Start Check Consistency on Virtual Drive 0 (target id: 0) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Start Check Consistency on Virtual Drive 1 (target id: 1) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Start Check Consistency on Virtual Drive 2 (target id: 2) Success.<br />
<br />
Exit Code: 0x00<br />
<br />
Check Consistency<br />
<br />
Progress of Virtual Drives...<br />
<br />
Virtual Drive # Percent Complete Time Elps<br />
0 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:03<br />
1 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:02<br />
2 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:01<br />
<br />
Press <ESC> key to quit...<br />
</pre><br />
<br />
The progress for each drive is displayed until all drives have completed the verify. We just want to make sure that each drive goes to completion. No followup is needed...though there probably is a log or history where we can get more info.<br />
<br />
You will notice that jail7 does not run a verify- that's on purpose. The last time we tried this it crashed the system. So, this must be run from the BIOS (take the system offline for a couple hours).<br />
<br />
See [[RAIC_CLI#LSI|LSI RAID CLI Reference]] for more details on how to use the CLI<br />
<br />
=== LSI-based controllers (megaraid) ===<br />
<br />
There is a CLI for this however it's easier to do this with a curses GUI app: megaraid<br />
<br />
Currently only on these servers: virt15, virt16, and firewall2<br />
<br />
To run:<br />
<br />
# cd /usr/local/sbin/; megamgr<br />
<br />
Main menu:<br />
<pre>²ÚÄÄManagement MenuÄÄ¿²<br />
²³ Configure ³²<br />
²³ Initialize ³²<br />
²³ Objects ³²<br />
²³ Rebuild ³²<br />
²³ Check Consistency ³²<br />
²³ Advanced Menu ³²<br />
²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²</pre><br />
<br />
Before you check consistency, make sure the arrays are healthy.<br />
<br />
Objects -> Physical Drive<br />
<br />
Then look to make sure they're all <tt>ONLIN</tt><br />
<pre><br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄObjects - PHYSICAL DRIVE SELECTION MENUÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²<br />
²²²²²²²²³ ³²²²²²²<br />
²²²²²²²²³ Channel-1 ³²²²²²²<br />
²ÚÄÄMana³ ID ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»x ³²²²²²²<br />
²³ Confi³ 0º* ONLIN A01-01º ³²²²²²²<br />
²³ Initi³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²³ Objec³ 1º* ONLIN A01-02º ³²²²²²²<br />
²³ Rebui³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²³ Check³ 2º* ONLIN A02-01º ³²²²²²²<br />
²³ Advan³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²ÀÄÄÄÄÄij 3º* ONLIN A02-02º ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 4º* ONLIN A03-01º ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 5º* ONLIN A03-02ºþ ³²²²²²²<br />
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍ͹ ³²²²²²²<br />
²²²²²²²²³ 6º* º ³²²²²²²<br />
²²²²²²²²³ ÈÍÍÍÍÍÍÍÍÍÍÍÍÍͼx ³²²²²²²<br />
²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²<br />
²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²²²<br />
²²²²²²²²³Ch-1 ID-5 DISK 140013MB SEAGATE ST3146707LC 0003 ³²²²²²²²²<br />
²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²<br />
</pre><br />
<br />
<br />
Once that's done, hit escape once then the back arror to move back to the Objects menu.<br />
So you select Objects -> Logical Drive -> Logical Drive 1 -> Check Consistency -> YES<br />
<br />
<pre><br />
<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÚÄLogical Drives(02)Ä¿²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ Logical Drive 1 ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²ÚÄÄÄÄObjectsÄÄij Logical Drive 2 ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²ÚÄÄManagemen³ Adapter ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Configure ³ Logical Drive ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Initialize³ Physical Drive ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Objects ³ Channel ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Rebuild ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Check Consistency ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²³ Advanced Menu ³²²²²²²²ÚÄÄÄÄLogical Drive 1ÄÄÄÄÄ¿²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²³ Initialize ÚÄCheck Consistency-1 ?Ä¿²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ Check Consiste³ YES ³²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ View/Update Pa³ NO ³²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³Select YES Or NO³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²<br />
</pre><br />
<br />
Then watch the progress. When done, escape back to Logical Drive then repeat for Logical Drive 2. If you ^C or accidentally escape out, you can come back in running the same commands and watch the progress again (it won't restart).<br />
<br />
You can exit megamgr by escaping out or ^C<br />
<br />
=== 3ware ===<br />
<br />
We are using 3ware controllers on backup1 & backup2. Running the verify script will give you different output based on the type of controller:<br />
<br />
<pre><br />
backup2 /d2# sh /root/verify.sh<br />
Controller: c0<br />
-------------<br />
Driver: 1.50.01.002<br />
Model: 7500-8<br />
FW: FE7X 1.05.00.068<br />
BIOS: BE7X 1.08.00.048<br />
Monitor: ME7X 1.01.00.040<br />
Serial #: F11605A3180172<br />
PCB: Rev3<br />
PCHIP: 1.30-33<br />
ACHIP: 3.20<br />
<br />
<br />
# of units: 3<br />
Unit 0: JBOD 186.31 GB ( 390721968 blocks): OK<br />
Unit 1: RAID 5 465.77 GB ( 976790016 blocks): DEGRADED<br />
Unit 5: RAID 5 698.65 GB ( 1465185024 blocks): DEGRADED<br />
<br />
# of ports: 8<br />
Port 0: WDC WD2000JB-00KFA0 WD-WCAMT1451690 186.31 GB (390721968 blocks): OK(unit 0)<br />
Port 1: WDC WD2500JB-00GVC0 WD-WCAL78219488 232.88 GB (488397168 blocks): OK(unit 1)<br />
Port 2: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT)<br />
Port 3: WDC WD2500JB-00GVC0 WD-WMAL73882417 232.88 GB (488397168 blocks): OK(unit 1)<br />
Port 4: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT)<br />
Port 5: WDC WD2500JB-00GVA0 WD-WMAL71338097 232.88 GB (488397168 blocks): OK(unit 5)<br />
Port 6: WDC WD2500JB-32EVA0 WD-WMAEH1301595 232.88 GB (488397168 blocks): OK(unit 5)<br />
Port 7: WDC WD2500JB-00GVC0 WD-WCAL78165566 232.88 GB (488397168 blocks): OK(unit 5)<br />
Controller: c1<br />
-------------<br />
Driver: 1.50.01.002<br />
Model: 7500-8<br />
FW: FE7X 1.05.00.068<br />
BIOS: BE7X 1.08.00.048<br />
Monitor: ME7X 1.01.00.040<br />
Serial #: F11605A3180167<br />
PCB: Rev3<br />
PCHIP: 1.30-33<br />
ACHIP: 3.20<br />
<br />
<br />
# of units: 2<br />
Unit 0: RAID 5 698.65 GB ( 1465185024 blocks): OK<br />
Unit 4: RAID 5 698.65 GB ( 1465185024 blocks): OK<br />
<br />
# of ports: 8<br />
Port 0: WDC WD2500JB-00GVA0 WD-WMAL71301258 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 1: WDC WD2500JB-00GVA0 WD-WMAL71322705 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 2: WDC WD2500JB-00GVA0 WD-WMAL71945050 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 3: WDC WD2500JB-00GVA0 WD-WMAL71316201 232.88 GB (488397168 blocks): OK(unit 0)<br />
Port 4: WDC WD2500JB-00GVC0 WD-WCAL78323749 232.88 GB (488397168 blocks): OK(unit 4)<br />
Port 5: WDC WD3200AAJB-00J3A0 WD-WCAV2V689068 298.09 GB (625142448 blocks): OK(unit 4)<br />
Port 6: WDC WD2500JB-00GVC0 WD-WCAL78234420 232.88 GB (488397168 blocks): OK(unit 4)<br />
Port 7: WDC WD2500JB-00GVC0 WD-WCAL78592213 232.88 GB (488397168 blocks): OK(unit 4)<br />
backup2 /d2#<br />
</pre><br />
<br />
On backup2 look for all ok, no verify.<br />
<br />
<pre><br />
[root@backup3 ~]# sh /root/verify.sh<br />
/c2 Driver Version = 1.26.02.002<br />
/c2 Model = 8006-2LP<br />
/c2 Available Memory = 512KB<br />
/c2 Firmware Version = FE8S 1.05.00.068<br />
/c2 Bios Version = BE7X 1.08.00.048<br />
/c2 Boot Loader Version = ME7X 1.01.00.040<br />
/c2 Serial Number = L018501C6481395<br />
/c2 PCB Version = Rev5<br />
/c2 PCHIP Version = 1.30-66<br />
/c2 ACHIP Version = 3.20<br />
/c2 Total Optimal Units = 1<br />
/c2 Not Optimal Units = 0<br />
<br />
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u1 RAID-1 OK - - - 931.512 ON -<br />
<br />
Port Status Unit Size Blocks Serial<br />
---------------------------------------------------------------<br />
p0 OK u1 931.51 GB 1953525168 WD-WMAW31148820<br />
p1 OK u1 931.51 GB 1953525168 WD-WCATR0277515<br />
<br />
<br />
Ctl Date Severity Alarm Message<br />
------------------------------------------------------------------------------<br />
<br />
Sending start verify message to /c2/u1 ... Done.<br />
<br />
<br />
when done run:<br />
tw_cli /c2 show alarms<br />
<br />
[root@backup3 ~]#<br />
</pre><br />
<br />
Automatically starts the verify, just run <tt>tw_cli /c2 show alarms</tt> as instructed to see the results of the verify.<br />
<br />
<pre><br />
[root@backup1 /data/deprecated]# sh /root/verify.sh<br />
/c0 Driver Version = 2.26.02.010<br />
/c0 Model = 9650SE-8LPML<br />
/c0 Available Memory = 224MB<br />
/c0 Firmware Version = FE9X 4.06.00.004<br />
/c0 Bios Version = BE9X 4.05.00.015<br />
/c0 Boot Loader Version = BL9X 3.08.00.001<br />
/c0 Serial Number = L326025A8270177<br />
/c0 PCB Version = Rev 032<br />
/c0 PCHIP Version = 2.00<br />
/c0 ACHIP Version = 1.90<br />
/c0 Number of Ports = 8<br />
/c0 Number of Drives = 6<br />
/c0 Number of Units = 1<br />
/c0 Total Optimal Units = 1<br />
/c0 Not Optimal Units = 0<br />
/c0 JBOD Export Policy = off<br />
/c0 Disk Spinup Policy = 1<br />
/c0 Spinup Stagger Time Policy (sec) = 1<br />
/c0 Auto-Carving Policy = off<br />
/c0 Auto-Carving Size = 2048 GB<br />
/c0 Auto-Rebuild Policy = on<br />
/c0 Controller Bus Type = PCIe<br />
/c0 Controller Bus Width = 1 lane<br />
/c0 Controller Bus Speed = 2.5 Gbps/lane<br />
<br />
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u0 RAID-5 OK - - 64K 4656.56 ON ON<br />
<br />
Port Status Unit Size Blocks Serial<br />
---------------------------------------------------------------<br />
p0 OK u0 931.51 GB 1953525168 9QJ1Y017<br />
p1 DEVICE-ERROR u0 931.51 GB 1953525168 9QJ1ZN07<br />
p2 OK u0 931.51 GB 1953525168 9QJ2XK1R<br />
p3 OK u0 931.51 GB 1953525168 9QJ2010B<br />
p4 OK u0 1.36 TB 2930277168 6XW0L36T<br />
p5 OK u0 931.51 GB 1953525168 WD-WMATV2444836<br />
p6 NOT-PRESENT - - - -<br />
p7 NOT-PRESENT - - - -<br />
<br />
<br />
Ctl Date Severity Alarm Message<br />
------------------------------------------------------------------------------<br />
c0 [Sat May 12 11:27:15 2012] WARNING Sector repair completed: port=0, LBA=0x6AE571C<br />
c0 [Sat May 12 19:16:21 2012] WARNING Sector repair completed: port=1, LBA=0x40E62A23<br />
c0 [Sat May 12 21:40:56 2012] INFO Verify completed: unit=0<br />
c0 [Mon May 14 00:53:53 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFA7<br />
c0 [Mon May 14 00:58:21 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAA<br />
c0 [Mon May 14 04:35:13 2012] WARNING Sector repair completed: port=0, LBA=0x8FEF2CF<br />
c0 [Mon May 14 04:38:22 2012] WARNING Sector repair completed: port=0, LBA=0x8FEF2D1<br />
c0 [Tue May 15 22:53:46 2012] WARNING Sector repair completed: port=0, LBA=0x13C2622<br />
c0 [Wed May 16 00:39:31 2012] WARNING Sector repair completed: port=0, LBA=0x365A67F<br />
c0 [Wed May 16 00:39:37 2012] WARNING Sector repair completed: port=0, LBA=0x365A685<br />
c0 [Wed May 16 00:47:18 2012] WARNING Sector repair completed: port=0, LBA=0x365A687<br />
c0 [Sat May 19 00:01:44 2012] INFO Verify started: unit=0<br />
c0 [Sat May 19 04:46:20 2012] WARNING Sector repair completed: port=0, LBA=0x365A68E<br />
c0 [Sat May 19 13:37:06 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAC<br />
c0 [Sat May 19 13:37:28 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAE<br />
c0 [Sat May 19 13:37:47 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFB1<br />
c0 [Sat May 19 13:38:00 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFB3<br />
c0 [Sat May 19 21:47:45 2012] INFO Verify completed: unit=0<br />
c0 [Wed May 23 12:21:41 2012] INFO Cache synchronization completed: unit=0<br />
c0 [Fri May 25 00:08:19 2012] WARNING Sector repair completed: port=0, LBA=0x12DA76C<br />
c0 [Fri May 25 00:08:34 2012] WARNING Sector repair completed: port=0, LBA=0x12E4901<br />
c0 [Fri May 25 00:09:33 2012] WARNING Sector repair completed: port=0, LBA=0x12DA773<br />
c0 [Fri May 25 00:39:12 2012] WARNING Sector repair completed: port=0, LBA=0x42C597B<br />
c0 [Sat May 26 00:01:45 2012] INFO Verify started: unit=0<br />
c0 [Sat May 26 00:42:05 2012] WARNING Sector repair completed: port=1, LBA=0x323C1AC<br />
c0 [Sat May 26 00:51:43 2012] WARNING Sector repair completed: port=1, LBA=0x323C1AE<br />
c0 [Sat May 26 01:54:21 2012] WARNING Sector repair completed: port=1, LBA=0x2F0D302<br />
c0 [Sat May 26 02:06:38 2012] WARNING Sector repair completed: port=0, LBA=0x12DA777<br />
c0 [Sat May 26 02:07:21 2012] WARNING Sector repair completed: port=0, LBA=0x12E48FE<br />
c0 [Sat May 26 04:20:00 2012] WARNING Sector repair completed: port=1, LBA=0x2F0D306<br />
c0 [Sat May 26 04:32:58 2012] WARNING Sector repair completed: port=1, LBA=0x323C1B1<br />
c0 [Sat May 26 04:33:21 2012] WARNING Sector repair completed: port=1, LBA=0x323C1B3<br />
c0 [Sat May 26 04:33:44 2012] WARNING Sector repair completed: port=1, LBA=0x323C1BA<br />
c0 [Sat May 26 05:24:07 2012] WARNING Sector repair completed: port=1, LBA=0x3F83862<br />
c0 [Sat May 26 05:25:09 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 06:08:13 2012] WARNING Sector repair completed: port=0, LBA=0x4CDC6A2<br />
c0 [Sat May 26 09:49:35 2012] WARNING Sector repair completed: port=1, LBA=0x6CACD4A<br />
c0 [Sat May 26 18:10:44 2012] WARNING Sector repair completed: port=1, LBA=0x18F425EA<br />
c0 [Sat May 26 19:45:40 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:22:52 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:15 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:22 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:35 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:41 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:49 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:23:57 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:24:02 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 20:54:41 2012] WARNING Verify fixed data/parity mismatch: unit=0<br />
c0 [Sat May 26 22:00:30 2012] INFO Verify completed: unit=0<br />
c0 [Sat Jun 2 00:01:43 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 2 00:30:17 2012] WARNING Sector repair completed: port=0, LBA=0x2B911E4<br />
c0 [Sat Jun 2 00:50:57 2012] WARNING Sector repair completed: port=0, LBA=0x5A807CA6<br />
c0 [Sat Jun 2 04:13:13 2012] WARNING Sector repair completed: port=0, LBA=0x2D18291<br />
c0 [Sat Jun 2 04:13:35 2012] WARNING Sector repair completed: port=0, LBA=0x2D1829F<br />
c0 [Sat Jun 2 21:48:02 2012] INFO Verify completed: unit=0<br />
c0 [Mon Jun 4 04:40:34 2012] WARNING Sector repair completed: port=1, LBA=0x4AF8098F<br />
c0 [Tue Jun 5 00:28:19 2012] WARNING Sector repair completed: port=1, LBA=0x263C5CD<br />
c0 [Tue Jun 5 00:33:06 2012] WARNING Sector repair completed: port=1, LBA=0x263C5CF<br />
c0 [Thu Jun 7 00:34:27 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B5F<br />
c0 [Thu Jun 7 00:38:50 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B61<br />
c0 [Fri Jun 8 00:07:13 2012] WARNING Sector repair completed: port=0, LBA=0xC131F6B<br />
c0 [Sat Jun 9 00:01:41 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 9 00:29:11 2012] WARNING Sector repair completed: port=0, LBA=0x6C7614D<br />
c0 [Sat Jun 9 00:38:25 2012] WARNING Sector repair completed: port=0, LBA=0x6C76152<br />
c0 [Sat Jun 9 04:02:30 2012] WARNING Sector repair completed: port=1, LBA=0x263C5D1<br />
c0 [Sat Jun 9 04:02:52 2012] WARNING Sector repair completed: port=1, LBA=0x263C5D3<br />
c0 [Sat Jun 9 04:07:32 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E12<br />
c0 [Sat Jun 9 04:07:57 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E15<br />
c0 [Sat Jun 9 04:08:16 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E17<br />
c0 [Sat Jun 9 04:08:45 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E19<br />
c0 [Sat Jun 9 04:15:04 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B64<br />
c0 [Sat Jun 9 04:15:26 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B66<br />
c0 [Sat Jun 9 04:15:45 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B68<br />
c0 [Sat Jun 9 04:15:59 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B6C<br />
c0 [Sat Jun 9 04:16:13 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B6E<br />
c0 [Sat Jun 9 21:48:52 2012] INFO Verify completed: unit=0<br />
c0 [Thu Jun 14 00:40:10 2012] WARNING Sector repair completed: port=0, LBA=0x334F14B<br />
c0 [Sat Jun 16 00:01:38 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 16 21:16:19 2012] INFO Verify completed: unit=0<br />
c0 [Tue Jun 19 02:03:43 2012] WARNING Sector repair completed: port=1, LBA=0xFE41EAD<br />
c0 [Wed Jun 20 02:30:02 2012] WARNING Sector repair completed: port=1, LBA=0xD99145C<br />
c0 [Sat Jun 23 00:01:36 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 23 04:27:04 2012] WARNING Sector repair completed: port=1, LBA=0x2FAD311<br />
c0 [Sat Jun 23 06:52:38 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC8D<br />
c0 [Sat Jun 23 06:53:03 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC91<br />
c0 [Sat Jun 23 06:53:21 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC94<br />
c0 [Sat Jun 23 17:00:22 2012] WARNING Sector repair completed: port=1, LBA=0xF9AC7C9<br />
c0 [Sat Jun 23 21:15:19 2012] INFO Verify completed: unit=0<br />
c0 [Sat Jun 30 00:01:34 2012] INFO Verify started: unit=0<br />
c0 [Sat Jun 30 05:24:13 2012] WARNING Sector repair completed: port=0, LBA=0x3FAA9E7<br />
c0 [Sat Jun 30 14:49:39 2012] WARNING Sector repair completed: port=1, LBA=0x869931C<br />
c0 [Sat Jun 30 21:31:05 2012] INFO Verify completed: unit=0<br />
c0 [Tue Jul 3 03:40:25 2012] WARNING Sector repair completed: port=1, LBA=0xD36C7F7<br />
c0 [Fri Jul 6 02:50:18 2012] WARNING Sector repair completed: port=1, LBA=0x3562470<br />
c0 [Fri Jul 6 22:18:26 2012] WARNING Sector repair completed: port=1, LBA=0x3563173<br />
c0 [Sat Jul 7 00:01:31 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 7 00:50:16 2012] WARNING Sector repair completed: port=0, LBA=0x76EE88<br />
c0 [Sat Jul 7 00:50:39 2012] WARNING Sector repair completed: port=0, LBA=0x76EE8F<br />
c0 [Sat Jul 7 21:39:36 2012] INFO Verify completed: unit=0<br />
c0 [Sun Jul 8 02:51:05 2012] WARNING Sector repair completed: port=0, LBA=0x67759D<br />
c0 [Sun Jul 8 02:53:55 2012] WARNING Sector repair completed: port=0, LBA=0x67759B<br />
c0 [Tue Jul 10 16:17:21 2012] WARNING Sector repair completed: port=0, LBA=0x15C8C695<br />
c0 [Wed Jul 11 22:51:22 2012] WARNING Sector repair completed: port=1, LBA=0x355BBD0<br />
c0 [Sat Jul 14 00:01:28 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 14 01:33:40 2012] WARNING Sector repair completed: port=1, LBA=0x1333BCF4<br />
c0 [Sat Jul 14 03:36:23 2012] WARNING Sector repair completed: port=1, LBA=0x2174773<br />
c0 [Sat Jul 14 11:26:44 2012] WARNING Sector repair completed: port=0, LBA=0x7429AB7<br />
c0 [Sat Jul 14 16:53:50 2012] WARNING Sector repair completed: port=1, LBA=0xA17EB3F<br />
c0 [Sat Jul 14 21:19:25 2012] INFO Verify completed: unit=0<br />
c0 [Wed Jul 18 05:08:47 2012] WARNING Sector repair completed: port=1, LBA=0x17D62EDC<br />
c0 [Wed Jul 18 05:14:15 2012] WARNING Sector repair completed: port=1, LBA=0x17D62EE1<br />
c0 [Thu Jul 19 03:24:59 2012] WARNING Sector repair completed: port=0, LBA=0x7733C3D<br />
c0 [Thu Jul 19 03:25:20 2012] WARNING Sector repair completed: port=0, LBA=0x773CEA5<br />
c0 [Thu Jul 19 03:28:16 2012] WARNING Sector repair completed: port=0, LBA=0x7733C42<br />
c0 [Thu Jul 19 03:28:41 2012] WARNING Sector repair completed: port=0, LBA=0x773CEAF<br />
c0 [Sat Jul 21 00:01:26 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 21 03:07:30 2012] WARNING Sector repair completed: port=1, LBA=0x1CC6936<br />
c0 [Sat Jul 21 03:07:52 2012] WARNING Sector repair completed: port=1, LBA=0x1CC6938<br />
c0 [Sat Jul 21 03:08:11 2012] WARNING Sector repair completed: port=1, LBA=0x1CC693A<br />
c0 [Sat Jul 21 16:43:56 2012] WARNING Sector repair completed: port=0, LBA=0xD04C914<br />
c0 [Sat Jul 21 16:45:31 2012] WARNING Sector repair completed: port=1, LBA=0xD456973<br />
c0 [Sat Jul 21 21:14:29 2012] INFO Verify completed: unit=0<br />
c0 [Wed Jul 25 03:37:25 2012] WARNING Sector repair completed: port=0, LBA=0x1F8E6C43<br />
c0 [Sat Jul 28 00:01:24 2012] INFO Verify started: unit=0<br />
c0 [Sat Jul 28 01:45:27 2012] WARNING Sector repair completed: port=0, LBA=0x11584AD<br />
c0 [Sat Jul 28 18:54:25 2012] WARNING Sector repair completed: port=1, LBA=0x447C3E6C<br />
c0 [Sat Jul 28 21:13:46 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 1 03:20:11 2012] WARNING Sector repair completed: port=0, LBA=0x805FEF<br />
c0 [Fri Aug 3 00:50:03 2012] WARNING Sector repair completed: port=0, LBA=0xCED0ACA<br />
c0 [Sat Aug 4 00:01:22 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 4 00:52:51 2012] WARNING Sector repair completed: port=0, LBA=0x805FF3<br />
c0 [Sat Aug 4 00:53:14 2012] WARNING Sector repair completed: port=0, LBA=0x805FF5<br />
c0 [Sat Aug 4 00:53:33 2012] WARNING Sector repair completed: port=0, LBA=0x805FF7<br />
c0 [Sat Aug 4 00:53:47 2012] WARNING Sector repair completed: port=0, LBA=0x805FF9<br />
c0 [Sat Aug 4 00:54:00 2012] WARNING Sector repair completed: port=0, LBA=0x805FFB<br />
c0 [Sat Aug 4 00:54:14 2012] WARNING Sector repair completed: port=0, LBA=0x805FFD<br />
c0 [Sat Aug 4 00:54:27 2012] WARNING Sector repair completed: port=0, LBA=0x805FFF<br />
c0 [Sat Aug 4 04:43:12 2012] WARNING Sector repair completed: port=1, LBA=0x16974289<br />
c0 [Sat Aug 4 04:58:17 2012] WARNING Sector repair completed: port=1, LBA=0x1697428E<br />
c0 [Sat Aug 4 20:54:53 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 8 03:21:55 2012] ERROR Drive timeout detected: port=1<br />
c0 [Wed Aug 8 15:31:44 2012] WARNING Sector repair completed: port=0, LBA=0x1A366CD3<br />
c0 [Sat Aug 11 00:01:21 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 11 20:40:51 2012] INFO Verify completed: unit=0<br />
c0 [Thu Aug 16 05:10:55 2012] WARNING Sector repair completed: port=0, LBA=0x1C22593<br />
c0 [Sat Aug 18 00:01:18 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 18 03:00:20 2012] WARNING Sector repair completed: port=0, LBA=0x1C225A5<br />
c0 [Sat Aug 18 03:43:00 2012] WARNING Sector repair completed: port=1, LBA=0x23EE91E<br />
c0 [Sat Aug 18 03:43:23 2012] WARNING Sector repair completed: port=1, LBA=0x23EE920<br />
c0 [Sat Aug 18 17:00:06 2012] WARNING Sector repair completed: port=1, LBA=0x137D066A<br />
c0 [Sat Aug 18 17:00:29 2012] WARNING Sector repair completed: port=1, LBA=0x137D066D<br />
c0 [Sat Aug 18 21:13:01 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 22 01:36:08 2012] WARNING Sector repair completed: port=0, LBA=0x2560A0F<br />
c0 [Wed Aug 22 01:37:42 2012] WARNING Sector repair completed: port=0, LBA=0x2560A13<br />
c0 [Fri Aug 24 04:01:36 2012] WARNING Sector repair completed: port=1, LBA=0x55C1A5DF<br />
c0 [Fri Aug 24 05:02:06 2012] WARNING Sector repair completed: port=1, LBA=0xCE3378A<br />
c0 [Sat Aug 25 00:01:17 2012] INFO Verify started: unit=0<br />
c0 [Sat Aug 25 00:31:06 2012] WARNING Sector repair completed: port=1, LBA=0x50F65D<br />
c0 [Sat Aug 25 00:39:52 2012] WARNING Sector repair completed: port=0, LBA=0x678FF4<br />
c0 [Sat Aug 25 03:43:15 2012] WARNING Sector repair completed: port=0, LBA=0x2560A15<br />
c0 [Sat Aug 25 03:43:39 2012] WARNING Sector repair completed: port=0, LBA=0x2560A19<br />
c0 [Sat Aug 25 03:43:58 2012] WARNING Sector repair completed: port=0, LBA=0x2560A1B<br />
c0 [Sat Aug 25 03:44:30 2012] WARNING Sector repair completed: port=0, LBA=0x2560A21<br />
c0 [Sat Aug 25 20:58:14 2012] INFO Verify completed: unit=0<br />
c0 [Wed Aug 29 04:57:15 2012] WARNING Sector repair completed: port=1, LBA=0xF3957EB<br />
c0 [Sat Sep 1 00:01:15 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 1 03:21:52 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC86<br />
c0 [Sat Sep 1 03:22:15 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC88<br />
c0 [Sat Sep 1 03:22:34 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC8A<br />
c0 [Sat Sep 1 03:22:47 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC8C<br />
c0 [Sat Sep 1 17:17:22 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD1<br />
c0 [Sat Sep 1 17:17:45 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD3<br />
c0 [Sat Sep 1 17:18:04 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD5<br />
c0 [Sat Sep 1 21:36:56 2012] INFO Verify completed: unit=0<br />
c0 [Thu Sep 6 00:07:30 2012] WARNING Sector repair completed: port=0, LBA=0xDA3C64B<br />
c0 [Thu Sep 6 00:32:56 2012] WARNING Sector repair completed: port=1, LBA=0x6BBA816<br />
c0 [Sat Sep 8 00:01:13 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 8 00:09:56 2012] WARNING Sector repair completed: port=0, LBA=0xDEBC958<br />
c0 [Sat Sep 8 04:38:45 2012] WARNING Sector repair completed: port=0, LBA=0x38D254F<br />
c0 [Sat Sep 8 20:44:50 2012] INFO Verify completed: unit=0<br />
c0 [Mon Sep 10 01:26:34 2012] WARNING Sector repair completed: port=1, LBA=0xFFD8D5E<br />
c0 [Wed Sep 12 00:33:48 2012] WARNING Sector repair completed: port=1, LBA=0xE8DB928<br />
c0 [Wed Sep 12 00:36:33 2012] WARNING Sector repair completed: port=1, LBA=0x6D49411<br />
c0 [Fri Sep 14 01:59:39 2012] WARNING Sector repair completed: port=0, LBA=0x1467F1C<br />
c0 [Fri Sep 14 02:08:27 2012] WARNING Sector repair completed: port=1, LBA=0x14C8ABD<br />
c0 [Fri Sep 14 03:54:47 2012] WARNING Sector repair completed: port=0, LBA=0x1580C915<br />
c0 [Sat Sep 15 00:01:11 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 15 02:38:14 2012] WARNING Sector repair completed: port=0, LBA=0x1C178973<br />
c0 [Sat Sep 15 02:59:02 2012] WARNING Sector repair completed: port=0, LBA=0x1C178975<br />
c0 [Sat Sep 15 04:47:08 2012] WARNING Sector repair completed: port=0, LBA=0x3FA0356<br />
c0 [Sat Sep 15 04:47:31 2012] WARNING Sector repair completed: port=0, LBA=0x3FA0359<br />
c0 [Sat Sep 15 10:41:59 2012] WARNING Sector repair completed: port=0, LBA=0x6DFD1EC<br />
c0 [Sat Sep 15 13:25:23 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD100<br />
c0 [Sat Sep 15 13:25:31 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD104<br />
c0 [Sat Sep 15 13:25:54 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD106<br />
c0 [Sat Sep 15 17:10:50 2012] WARNING Sector repair completed: port=0, LBA=0x1C178977<br />
c0 [Sat Sep 15 20:59:57 2012] INFO Verify completed: unit=0<br />
c0 [Tue Sep 18 01:17:18 2012] WARNING Sector repair completed: port=1, LBA=0x803B05B<br />
c0 [Sat Sep 22 00:01:10 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 22 20:54:31 2012] INFO Verify completed: unit=0<br />
c0 [Tue Sep 25 01:56:47 2012] WARNING Sector repair completed: port=0, LBA=0x26E3909<br />
c0 [Sat Sep 29 00:01:08 2012] INFO Verify started: unit=0<br />
c0 [Sat Sep 29 02:04:14 2012] WARNING Sector repair completed: port=0, LBA=0x146AC03<br />
c0 [Sat Sep 29 10:58:39 2012] WARNING Sector repair completed: port=0, LBA=0x6D4EB0E<br />
c0 [Sat Sep 29 10:59:02 2012] WARNING Sector repair completed: port=0, LBA=0x6D4EB14<br />
c0 [Sat Sep 29 11:22:44 2012] WARNING Sector repair completed: port=0, LBA=0x6F79623<br />
c0 [Sat Sep 29 13:50:48 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D65E<br />
c0 [Sat Sep 29 13:51:11 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D661<br />
c0 [Sat Sep 29 13:51:30 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D663<br />
c0 [Sat Sep 29 20:57:34 2012] INFO Verify completed: unit=0<br />
c0 [Mon Oct 1 04:47:24 2012] WARNING Sector repair completed: port=0, LBA=0xC5BC6F2<br />
c0 [Tue Oct 2 02:00:27 2012] WARNING Sector repair completed: port=0, LBA=0x1547667<br />
c0 [Tue Oct 2 02:01:56 2012] WARNING Sector repair completed: port=0, LBA=0x154766F<br />
c0 [Tue Oct 2 05:02:31 2012] WARNING Sector repair completed: port=1, LBA=0xD67D054<br />
c0 [Tue Oct 2 05:04:14 2012] WARNING Sector repair completed: port=1, LBA=0xD67D056<br />
c0 [Wed Oct 3 01:22:12 2012] WARNING Sector repair completed: port=1, LBA=0x12AAF8CA<br />
c0 [Thu Oct 4 04:29:22 2012] WARNING Sector repair completed: port=0, LBA=0x13E6F992<br />
c0 [Thu Oct 4 05:10:51 2012] WARNING Sector repair completed: port=0, LBA=0x1C252A4<br />
c0 [Sat Oct 6 00:01:07 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 6 19:41:18 2012] WARNING Sector repair completed: port=1, LBA=0x5A5C3AE8<br />
c0 [Sat Oct 6 21:01:05 2012] INFO Verify completed: unit=0<br />
c0 [Mon Oct 8 00:32:06 2012] WARNING Sector repair completed: port=0, LBA=0x6C60D3E<br />
c0 [Tue Oct 9 03:51:03 2012] WARNING Sector repair completed: port=1, LBA=0x89B5EC9<br />
c0 [Thu Oct 11 04:21:17 2012] WARNING Sector repair completed: port=1, LBA=0x13F85833<br />
c0 [Sat Oct 13 00:01:05 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 13 05:12:40 2012] WARNING Sector repair completed: port=0, LBA=0x3FA5134<br />
c0 [Sat Oct 13 21:08:35 2012] INFO Verify completed: unit=0<br />
c0 [Tue Oct 16 03:53:50 2012] WARNING Sector repair completed: port=1, LBA=0x148AA1BD<br />
c0 [Thu Oct 18 03:20:30 2012] WARNING Sector repair completed: port=0, LBA=0x1C8DABCB<br />
c0 [Thu Oct 18 04:52:50 2012] WARNING Sector repair completed: port=0, LBA=0xE879057<br />
c0 [Sat Oct 20 00:01:04 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 20 02:19:25 2012] WARNING Sector repair completed: port=1, LBA=0x174B012<br />
c0 [Sat Oct 20 03:41:38 2012] WARNING Sector repair completed: port=0, LBA=0x256D93B<br />
c0 [Sat Oct 20 03:42:01 2012] WARNING Sector repair completed: port=0, LBA=0x256D93D<br />
c0 [Sat Oct 20 03:42:40 2012] WARNING Sector repair completed: port=0, LBA=0x256D940<br />
c0 [Sat Oct 20 03:42:59 2012] WARNING Sector repair completed: port=0, LBA=0x256D942<br />
c0 [Sat Oct 20 03:43:12 2012] WARNING Sector repair completed: port=0, LBA=0x256D944<br />
c0 [Sat Oct 20 03:43:26 2012] WARNING Sector repair completed: port=0, LBA=0x256D948<br />
c0 [Sat Oct 20 16:37:52 2012] WARNING Sector repair completed: port=0, LBA=0xE879060<br />
c0 [Sat Oct 20 16:38:15 2012] WARNING Sector repair completed: port=0, LBA=0xE879062<br />
c0 [Sat Oct 20 21:00:18 2012] INFO Verify completed: unit=0<br />
c0 [Sat Oct 20 23:49:01 2012] WARNING Sector repair completed: port=1, LBA=0x4473E908<br />
c0 [Sun Oct 21 03:42:26 2012] WARNING Sector repair completed: port=1, LBA=0x175BADD5<br />
c0 [Tue Oct 23 01:09:04 2012] WARNING Sector repair completed: port=1, LBA=0x6E524860<br />
c0 [Fri Oct 26 03:21:25 2012] WARNING Sector repair completed: port=0, LBA=0x802C61<br />
c0 [Fri Oct 26 04:22:21 2012] WARNING Sector repair completed: port=0, LBA=0x176353CD<br />
c0 [Sat Oct 27 00:01:03 2012] INFO Verify started: unit=0<br />
c0 [Sat Oct 27 00:49:35 2012] WARNING Sector repair completed: port=0, LBA=0x802C65<br />
c0 [Sat Oct 27 17:02:24 2012] WARNING Sector repair completed: port=1, LBA=0xC1FF26D<br />
c0 [Sat Oct 27 17:09:06 2012] WARNING Sector repair completed: port=0, LBA=0xDF621AD<br />
c0 [Sat Oct 27 21:30:57 2012] INFO Verify completed: unit=0<br />
c0 [Tue Oct 30 00:20:46 2012] WARNING Sector repair completed: port=0, LBA=0xE9FE2AB<br />
c0 [Wed Oct 31 02:02:03 2012] WARNING Sector repair completed: port=0, LBA=0x1460C25<br />
c0 [Wed Oct 31 02:04:05 2012] WARNING Sector repair completed: port=0, LBA=0x1460C28<br />
c0 [Thu Nov 1 00:48:34 2012] WARNING Sector repair completed: port=1, LBA=0xA7C92BE<br />
c0 [Thu Nov 1 05:04:45 2012] WARNING Sector repair completed: port=0, LBA=0x1C252C2<br />
<br />
[root@backup1 /data/deprecated]#<br />
</pre><br />
<br />
Look for failed drives and errors. Obviously from the above we need to probably replace drives 0 and 1 and drive 1 is even showing as having problems, yet the RAID array is healthy, amazingly. You also see the automatic verifies.<br />
<br />
Note: when rebuilding a degraded mirror, you will see no progress as it rebuilds in the cli<br />
<pre>Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<br />
------------------------------------------------------------------------------<br />
u0 RAID-5 REBUILDING 0 - 64K 4656.56 OFF ON</pre><br />
<br />
<br />
=== areca ===<br />
<br />
We are using an areca controller on backup3. <br />
<br />
<pre><br />
[root@newbackup3 ~]# sh /root/verify.sh<br />
# Name Raid Name Level Capacity Ch/Id/Lun State<br />
===============================================================================<br />
1 ARC-1160-VOL#00 Raid Set # 00 Raid5 5000.0GB 00/00/00 Checking(19.7%)<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
# Name Disks TotalCap FreeCap DiskChannels State<br />
===============================================================================<br />
1 Raid Set # 00 6 6000.0GB 0.0GB 123456 Checking<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
Date-Time Device Event Type Elapsed Time Errors<br />
===============================================================================<br />
2012-12-05 20:40:58 ARC-1160-VOL#00 Start Checking<br />
2012-12-01 05:06:04 ARC-1160-VOL#00 Complete Init 027:30:45<br />
2012-11-30 01:35:19 ARC-1160-VOL#00 Start Initialize<br />
2026-08-06 01:34:52 H/W MONITOR Raid Powered On<br />
2012-11-30 01:33:36 ARC-1160-VOL#00 Stop Initialization 000:31:48<br />
2012-11-30 01:01:47 ARC-1160-VOL#00 Start Initialize<br />
2026-08-06 00:58:13 H/W MONITOR Raid Powered On<br />
2012-11-30 00:57:26 ARC-1160-VOL#00 Stop Initialization 000:57:07<br />
2012-11-30 00:00:19 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 23:56:48 H/W MONITOR Raid Powered On<br />
2026-08-05 23:52:58 H/W MONITOR Raid Powered On<br />
2026-08-05 23:50:14 H/W MONITOR Raid Powered On<br />
2026-08-05 23:43:30 H/W MONITOR Raid Powered On<br />
2012-11-29 23:10:07 ARC-1160-VOL#00 Stop Initialization 000:00:56<br />
2012-11-29 23:09:11 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 23:08:57 H/W MONITOR Raid Powered On<br />
2012-11-29 23:08:10 ARC-1160-VOL#00 Stop Initialization 000:20:41<br />
2012-11-29 22:47:29 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 22:46:59 H/W MONITOR Raid Powered On<br />
2026-08-05 22:45:55 H/W MONITOR Raid Powered On<br />
2026-08-05 22:44:53 H/W MONITOR Raid Powered On<br />
2026-08-05 22:42:06 H/W MONITOR Raid Powered On<br />
2026-08-05 22:40:50 H/W MONITOR Raid Powered On<br />
2012-11-29 22:40:04 ARC-1160-VOL#00 Stop Initialization 000:24:25<br />
2012-11-29 22:15:38 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 22:15:11 000:000001215B00 Restart Init LBA Point<br />
2026-08-05 22:15:10 H/W MONITOR Raid Powered On<br />
2012-11-29 21:56:38 ARC-1160-VOL#00 Start Initialize<br />
2026-08-05 21:56:12 H/W MONITOR Raid Powered On<br />
2026-08-05 21:56:04 IDE Channel #03 Device Inserted<br />
2012-11-29 21:55:13 IDE Channel #04 Device Inserted<br />
2012-11-29 21:55:03 IDE Channel #02 Device Inserted<br />
2026-08-05 21:53:09 H/W MONITOR Raid Powered On<br />
2026-08-05 20:51:46 H/W MONITOR Raid Powered On<br />
2026-08-05 20:49:56 H/W MONITOR Raid Powered On<br />
2026-08-05 20:48:29 H/W MONITOR Raid Powered On<br />
2026-08-05 20:46:29 H/W MONITOR Raid Powered On<br />
2026-08-05 20:44:49 H/W MONITOR Raid Powered On<br />
2026-08-05 20:43:01 H/W MONITOR Raid Powered On<br />
2026-08-05 20:36:25 H/W MONITOR Raid Powered On<br />
2026-08-05 20:31:18 H/W MONITOR Raid Powered On<br />
2026-08-05 20:30:08 H/W MONITOR Raid Powered On<br />
2026-08-05 20:08:40 H/W MONITOR Raid Powered On<br />
2026-08-05 20:06:11 H/W MONITOR Raid Powered On<br />
2026-08-05 20:05:14 H/W MONITOR Raid Powered On<br />
2026-08-05 20:03:58 H/W MONITOR Raid Powered On<br />
2026-08-05 20:00:56 H/W MONITOR Raid Powered On<br />
2026-08-05 19:57:57 H/W MONITOR Raid Powered On<br />
2026-08-05 19:56:15 H/W MONITOR Raid Powered On<br />
2026-08-05 19:55:05 H/W MONITOR Raid Powered On<br />
2026-08-05 17:24:36 H/W MONITOR Raid Powered On<br />
2026-08-05 17:22:43 H/W MONITOR Raid Powered On<br />
2026-08-05 04:50:42 H/W MONITOR Raid Powered On<br />
2026-08-05 04:47:33 H/W MONITOR Raid Powered On<br />
2026-08-05 04:43:57 H/W MONITOR Raid Powered On<br />
2026-08-05 04:18:52 H/W MONITOR Raid Powered On<br />
2026-08-05 04:17:30 H/W MONITOR Raid Powered On<br />
2026-08-05 04:13:30 H/W MONITOR Raid Powered On<br />
2026-08-05 04:10:26 H/W MONITOR Raid Powered On<br />
2026-08-05 04:09:23 H/W MONITOR Raid Powered On<br />
2026-08-05 00:08:09 H/W MONITOR Raid Powered On<br />
2026-08-05 00:07:12 H/W MONITOR Raid Powered On<br />
2026-08-05 00:05:51 H/W MONITOR Raid Powered On<br />
2026-08-05 00:04:27 H/W MONITOR Raid Powered On<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
press enter when ready to run verify<br />
</pre><br />
<br />
Look for failed drives and errors. <br />
<br />
When it proceed's to verifying, you can confirm with:<br />
<br />
<pre><br />
[root@newbackup3 ~]# cli64 vsf info<br />
# Name Raid Name Level Capacity Ch/Id/Lun State<br />
===============================================================================<br />
1 ARC-1160-VOL#00 Raid Set # 00 Raid5 5000.0GB 00/00/00 Checking(22.5%)<br />
===============================================================================<br />
GuiErrMsg<0x00>: Success.<br />
[root@newbackup3 ~]#<br />
</pre><br />
<br />
== Update OS list ==<br />
<br />
# check for any new VZ templates we want to offer: <tt>vzup2date -z</tt><br />
# see if there's any OS's we want to include in our colo install list. Update 2 places: <tt>signup/html/colo_quote.html</tt> & <tt>signup/html/step1.html</tt><br />
# update the mgmt database (ref_templates table, ref_systems table).<br />
<br />
= Infrequent tasks =<br />
<br />
== Free up space on gateway ==<br />
<br />
<pre>newgateway /var/spool# cd clientmqueue/<br />
newgateway /var/spool/clientmqueue# sh<br />
# for f in `ls`; do rm $f; done<br />
exit</pre><br />
<br />
== Free up space on mail ==<br />
<br />
You can clear out root mail:<br />
<pre><br />
mail /var/log# ll -h /var/mail/root<br />
-rw------- 1 root mail 543K Dec 19 13:05 /var/mail/root<br />
mail /var/log# rm /var/mail/root<br />
</pre><br />
<br />
Or you can archive mail logs:<br />
<pre><br />
mail /var/log# ls -l htt*<br />
-rw-r--r-- 1 root wheel 297436931 Dec 19 13:26 httpd-access.log<br />
-rw-r--r-- 1 root wheel 9824324 Jul 4 11:34 httpd-access.log.old.0.gz<br />
-rw-r--r-- 1 root wheel 6884137 Mar 17 2012 httpd-access.log.old.1.gz<br />
-rw-r--r-- 1 root wheel 18557444 Dec 3 2009 httpd-access.log.old.10.gz<br />
-rw-r--r-- 1 root wheel 14740263 Jan 9 2007 httpd-access.log.old.11.gz<br />
-rw-r--r-- 1 root wheel 14209465 Nov 28 2007 httpd-access.log.old.12.gz<br />
-rw-r--r-- 1 root wheel 16874396 Feb 19 2012 httpd-access.log.old.3.gz<br />
-rw-r--r-- 1 root wheel 14554859 Jul 22 2011 httpd-access.log.old.4.gz<br />
-rw-r--r-- 1 root wheel 10513227 Feb 18 2011 httpd-access.log.old.5.gz<br />
-rw-r--r-- 1 root wheel 7201946 Oct 29 2010 httpd-access.log.old.6.gz<br />
-rw-r--r-- 1 root wheel 10062537 May 6 2010 httpd-access.log.old.7.gz<br />
-rw-r--r-- 1 root wheel 10157042 Aug 12 2010 httpd-access.log.old.8.gz<br />
-rw-r--r-- 1 root wheel 11909534 Mar 4 2010 httpd-access.log.old.9.gz<br />
-rw-r--r-- 1 root wheel 59030930 Dec 19 13:01 httpd-error.log<br />
-rw-r--r-- 1 root wheel 3413134 Mar 4 2010 httpd-error.log.0.gz<br />
-rw-r--r-- 1 root wheel 795515 May 1 2007 httpd-error.log.1.gz<br />
-rw-r--r-- 1 root wheel 1142153 Nov 30 2007 httpd-error.log.2.gz<br />
-rw-r--r-- 1 root wheel 2325801 Feb 18 2011 httpd-error.log.gz<br />
<br />
mail /var/log# sh<br />
# for f in 12 11 10 9 8 7 6 5 4 3 2 1 0; do g=`echo $f+1|bc`; mv httpd-access.log.old.$f.gz httpd-access.log.old.$g.gz; done<br />
# mv httpd-access.log httpd-access.log.old.0<br />
# touch httpd-access.log<br />
# apachectl restart<br />
# gzip httpd-access.log.old.0<br />
<br />
# for f in 2 1 0; do g=`echo $f+1|bc`; mv httpd-error.log.$f.gz httpd-error.log.$g.gz; done<br />
# mv httpd-error.log httpd-error.log.0<br />
# touch httpd-error.log<br />
# apachectl restart<br />
# gzip httpd-error.log.0<br />
# exit<br />
</pre><br />
<br />
== Free up space on bwdb2 ==<br />
<br />
You can either remove items from <tt>/usr/home/archive</tt> or you can scp them to backup3:/data/bwdb2/archive .<br />
<br />
== Free up space on backup1 ==<br />
<br />
backup1 is our primary customer backup system. As usage grows over time, it needs to be regularly purged of old files. The easiest way to do this is by removing deprecated files. These mostly consist of cancelled customers or temporary dump/storage files (created during dump/restores). Our standard policy is to hang onto cancelled customers for 6mos after which we remove their files (as far as customers know their data is purged immediately, but we hang onto it just in case.. and in some cases we cancel a server due to non payment so this makes it easy to restore their system). To find files to remove:<br />
<br />
<pre>[root@backup1 ~]# cd /data/deprecated/<br />
[root@backup1 /data/deprecated]# ls<br />
2101-migrated-20120317.tgz old-683-cxld-20121021.tgz<br />
69.55.230.2-wwwbackup old-744-cxld-20120708.tgz<br />
991-DONTDELETE.tgz old-809-cxld-20120609.tgz<br />
archive-col02050-mdfile-cxld-20120409.gz old-854-cxld-20120621.tgz<br />
col01371.tgz old-931-cxld-20060513.tgz<br />
deleteme_ubuntu-10.10-x86_20111205 old-col00123-mdfile-noarchive-20120417.gz<br />
jail10_old old-col00147-vnfile-cxld-20120828.gz<br />
jail14_rsync_old old-col00419-dump-cxld-20120224.gz<br />
jail15_old old-col01098-vnfile-cxld-20120827.gz<br />
jail3_old old-col01278-dump-cxld-20120822<br />
jail4_old old-col01517-dump-cxld-20120828<br />
jail5_old old-col01669-dump-cxld-20120203.gz<br />
old-1009-cxld-20120608.tgz old-col01687-dump-cxld-20120909<br />
old-1012-cxld-20120411.tgz old-col01790-dump-cxld-20120828<br />
old-1052-cxld-20120721.tgz old-col01812-dump-cxld-20120820<br />
old-10631-cxld-20120622.tgz old-col01938-mdfile-cxld-20120619.gz<br />
old-10632-cxld-20120622.tgz old-col02095-mdfile-noarchive-20120523.gz<br />
old-10633-cxld-20120622.tgz olddebian-3.0-v15-20110610.tgz<br />
old-1236-cxld-20120621.tgz oldmod_frontpage-deb30-v15-20110610.tgz<br />
old-1381-cxld-20120404.tgz oldmod_perl-deb30-v15-20110610.tgz<br />
old-1422-cxld-20120721.tgz oldmod_ssl-deb30-v15-20110610.tgz<br />
old-14681-cxld-20120619.tgz oldmysql-deb30-v15-20110610.tgz<br />
old-1544-cxld-20120626.tgz oldproftpd-deb30-v15-20110610.tgz<br />
old-18351-cxld-20120605.tgz old_virt14<br />
old-1853-cxld-20120910.tgz old_virt18<br />
old-1963-cxld-20120206.tgz oldwebmin-deb30-v15-20110610.tgz<br />
old-1967-cxld-20120605.tgz suse.virt11.20120421.tgz<br />
old-1981-noarchive-20120729.tgz virt11<br />
old-2030-migrated-noarchive-20120727.tgz virt12_old<br />
old-2037-cxld-20120716.tgz virt13_old<br />
old-2065-cxld-20120727.tgz virt16_old<br />
old-2068-cxld-20120424.tgz virt4_old<br />
old-2085-cxld-20120531.tgz virt5_old<br />
old-364-cxld-20120904.tgz virt6_old<br />
old-446-cxld-20120512.tgz virt7_old<br />
old-613-cxld-20120601.tgz virt8_old<br />
[root@backup1 /data/deprecated]#<br />
</pre><br />
<br />
virtX_old and jailX_old are permanently archived, so ignore those as well as anything else marked not to delete or otherwise suspicious. Likewise, probably a good idea to try to hang onto oldTEMPLATE.gz as long as we can as well.<br />
Most of the stuff we want to delete is dated when it was deprecated, making this easy. So to remove files from 6 mos ago (running this in Oct):<br />
<br />
<pre>[root@backup1 /data/deprecated]# ls old*201204*<br />
old-1012-cxld-20120411.tgz old-2068-cxld-20120424.tgz<br />
old-1381-cxld-20120404.tgz old-col00123-mdfile-noarchive-20120417.gz<br />
[root@backup1 /data/deprecated]# rm old*201204*<br />
</pre><br />
<br />
Every few months you will also want to remove some of the snapshot archives for mail. We typically save the 1st, 10th, and 20th of each month. To do this you set aside the dates you want to save then remove months at a time, followed by restoring the set aside dates. Here's how that works:<br />
<pre>[root@backup1 /data/www/daily]# ls<br />
05 08-10-11 10-04-10 11-10-10 12-07-29 12-09-21 12-11-14<br />
06 08-10-21 10-04-20 11-10-20 12-07-30 12-09-22 12-11-15<br />
06-06-01-usr-home.tgz 08-11-01 10-05-01 11-11-01 12-07-31 12-09-23 12-11-16<br />
06-07-01-usr-home.tgz 08-11-10 10-05-11 11-11-10 12-08-01 12-09-24 12-11-17<br />
06-08-01-usr-home.tgz 08-11-20 10-05-20 11-11-20 12-08-02 12-09-25 12-11-18<br />
06-09-01-usr-home.tgz 08-12-01 10-06-01 11-12-01 12-08-03 12-09-26 12-11-19<br />
06-11-10 08-12-10 10-06-10 11-12-10 12-08-04 12-09-27 12-11-20<br />
06-12-21 08-12-20 10-06-20 11-12-20 12-08-05 12-09-28 12-11-21<br />
07-01-10 09-01-01 10-07-01 12-01-01 12-08-06 12-09-29 12-11-22<br />
07-01-20 09-01-10 10-07-10 12-01-10 12-08-07 12-09-30 12-11-23<br />
07-02-10 09-01-20 10-07-20 12-01-20 12-08-08 12-10-01 12-11-24<br />
07-02-20 09-02-01 10-08-01 12-02-01 12-08-09 12-10-02 12-11-25<br />
07-03-01 09-02-10 10-08-10 12-02-10 12-08-10 12-10-03 12-11-26<br />
07-03-20 09-02-20 10-08-20 12-02-20 12-08-11 12-10-04 12-11-27<br />
07-04-01 09-03-01 10-09-01 12-03-01 12-08-12 12-10-05 12-11-28<br />
07-04-10 09-03-10 10-09-10 12-03-10 12-08-13 12-10-06 12-11-29<br />
07-04-20 09-03-20 10-09-20 12-03-20 12-08-14 12-10-07 12-11-30<br />
07-05-01 09-04-01 10-10-01 12-04-01 12-08-15 12-10-08 12-12-01<br />
07-05-10 09-04-10 10-10-10 12-04-10 12-08-16 12-10-09 12-12-02<br />
07-05-20 09-04-20 10-10-20 12-04-20 12-08-17 12-10-10 12-12-03<br />
07-06-01 09-05-01 10-11-01 12-05-01 12-08-18 12-10-11 12-12-04<br />
07-06-10 09-05-10 10-11-10 12-05-10 12-08-19 12-10-12 12-12-05<br />
07-06-20 09-05-20 10-11-20 12-05-20 12-08-20 12-10-13 12-12-06<br />
07-07-20 09-06-01 10-12-01 12-06-01 12-08-21 12-10-14 12-12-07<br />
07-08-10 09-06-10 10-12-10 12-06-10 12-08-22 12-10-15 12-12-08<br />
07-08-20 09-06-20 10-12-20 12-06-20 12-08-23 12-10-16 12-12-09<br />
07-09-01 09-07-01 11-01-01 12-07-01 12-08-24 12-10-17 12-12-10<br />
07-10-01 09-07-10 11-01-10 12-07-02 12-08-25 12-10-18 12-12-11<br />
07-10-10 09-07-20 11-01-21 12-07-03 12-08-26 12-10-19 12-12-12<br />
07-10-20 09-08-01 11-02-01 12-07-04 12-08-27 12-10-20 12-12-13<br />
07-12-01 09-08-10 11-02-10 12-07-05 12-08-28 12-10-21 12-12-14<br />
07-12-10 09-08-20 11-02-20 12-07-06 12-08-29 12-10-22 12-12-15<br />
08-01-01 09-09-01 11-03-01 12-07-07 12-08-30 12-10-23 12-12-16<br />
08-01-20 09-09-10 11-03-10 12-07-08 12-08-31 12-10-24 12-12-17<br />
08-02-20 09-09-20 11-03-20 12-07-09 12-09-01 12-10-25 12-12-18<br />
08-03-01 09-10-01 11-04-01 12-07-10 12-09-02 12-10-26 12-12-19<br />
08-03-10 09-10-10 11-04-10 12-07-11 12-09-03 12-10-27 12-12-20<br />
08-03-20 09-10-20 11-04-20 12-07-12 12-09-04 12-10-28 12-12-21<br />
08-04-01 09-11-01 11-05-01 12-07-13 12-09-05 12-10-29 12-12-22<br />
08-04-20 09-11-10 11-05-10 12-07-14 12-09-06 12-10-30 12-12-23<br />
08-05-01 09-11-20 11-05-20 12-07-15 12-09-07 12-10-31 12-12-24<br />
08-05-10 09-12-01 11-06-01 12-07-16 12-09-08 12-11-01 12-12-25<br />
08-06-10 09-12-10 11-06-10 12-07-17 12-09-09 12-11-02 12-12-26<br />
08-06-20 09-12-20 11-06-20 12-07-18 12-09-10 12-11-03 12-12-27<br />
08-07-02 10-01-01 11-07-01 12-07-19 12-09-11 12-11-04 12-12-28<br />
08-07-10 10-01-10 11-07-10 12-07-20 12-09-12 12-11-05 2008-10-23<br />
08-07-20 10-01-20 11-07-20 12-07-21 12-09-13 12-11-06 bb.tgz<br />
08-08-01 10-02-01 11-08-01 12-07-22 12-09-14 12-11-07 boot<br />
08-08-10 10-02-10 11-08-10 12-07-23 12-09-15 12-11-08 current<br />
08-08-21 10-02-20 11-08-20 12-07-24 12-09-16 12-11-09 hold<br />
08-09-01 10-03-01 11-09-01 12-07-25 12-09-17 12-11-10<br />
08-09-10 10-03-10 11-09-10 12-07-26 12-09-18 12-11-11<br />
08-09-21 10-03-20 11-09-20 12-07-27 12-09-19 12-11-12<br />
08-10-01 10-04-01 11-10-01 12-07-28 12-09-20 12-11-13<br />
[root@backup1 /data/www/daily]#<br />
</pre><br />
<br />
So we see that everything up to July 2012 has been pruned. To prune July 2012 we do the following:<br />
<br />
<pre>mv 12-07-01 hold<br />
mv 12-07-10 hold<br />
mv 12-07-20 hold<br />
rm -fr 12-07*<br />
mv hold/* .</pre></div>75.80.128.147https://wiki.jcihosting.com/index.php?title=NetHere&diff=2199NetHere2020-02-21T20:41:53Z<p>75.80.128.147: /* NetHere */</p>
<hr />
<div>= NetHere =<br />
<br />
URL: http://toolbox.nethere.com<br />
<br />
<br><br />
Access to servers is restricted to admin-1.nethere.net. Access to admin-1.nethere.net is restricted to our office and nat.johncompanies.com.<br />
<br />
<br><br />
JCI Hosting acquired the web-hosting customers of NetHere. (Previously, NetHere had aquired customers from Simply Internet, and Z-net).<br />
Customer billing/info are being imported from NetHere to JCI database <br />
URL: https://secure.johncompanies.com/mgmt/index.html<br />
<br />
<br><br />
Nethere has several tools for management. These are restricted to only allow<br />
access from the office, or a few admin's homes.<br />
<br />
<br />
cacti [http://cacti.nethere.net]<br />
nagios[http://nagios.nethere.net]<br />
toolbox [http://toolbox.nethere.net]<br />
helpdesk [http://helpdesk.nethere.net]<br />
support FAQ [http://www.nethere.net/website/support]<br />
knowledge base [http://support.nethere.net/kb]<br />
phpmyadmin [https://phpmyadmin.nethere.net]<br />
<br />
<br />
== Virtual Hosts ==<br />
<br />
<pre><br />
<br />
66.63.129.2 - New HP DL360<br />
fpweb-2.nethere.net<br />
unixweb-6.nethere.net<br />
mta-db.mail.nethere.net<br />
scan-1.mail.nethere.net<br />
<br />
66.63.129.4- vmware9.eng<br />
mysqldb-1.webhost.nethere.net<br />
<br />
66.63.129.91<br />
lists-1.nethere.net<br />
sb-2.nethere.net<br />
mx-1.nethere.net- spooling server<br />
mailx-1.nethere.net- web server outbound mail server<br />
<br />
66.63.129.101 (c.host.nethere.net)<br />
ahi.nethere.net- ldap<br />
mailbox-4<br />
mta-2.mail.nethere.net<br />
relay-2.mail.nethere.net<br />
unixweb-8.nethere.net<br />
<br />
66.63.129.102 (vm2.eng.nethere.net)<br />
koi.nethere.net- off- realserver<br />
prov-1.nethere.net- cp.nethere.net, cp.znet.net, cp.simplyweb.net<br />
scribe.nethere.net- prov.nethere.net, scribe.nethere.net<br />
tetra.nethere.net- nethere.com, znet.com, simplyweb.com<br />
web-2.inboxhq.net- helpdesk<br />
webmail-1.nethere.net<br />
<br />
66.63.129.103<br />
andromeda.nethere.net (ns2.nethere.net)<br />
nscache-2.nethere.net<br />
nsrbl-2.nethere.net<br />
mailbox-2.nethere.net<br />
scan-2.mail.nethere.net<br />
<br />
66.63.129.104<br />
eel.nethere.net- ldap<br />
mta-1.mail.nethere.net<br />
nscache-1.nethere.net<br />
nsrbl-1.nethere.net<br />
phoenix.nethere.net (ns1.nethere.net)<br />
relay-1.mail.nethere.net<br />
unixweb-2.nethere.net<br />
<br />
66.63.129.105<br />
home-1.nethere.net<br />
ntweb-11.nethere.net<br />
ntweb-6.nethere.net<br />
cart32.nethere.net<br />
(web-3.schedulecafe.com)<br />
<br />
66.63.129.106 6499MB/16381 18.32GB/268.25 free<br />
admin-1.nethere.net<br />
admin-2.nethere.net<br />
pike.nethere.net- cacti, nagios- off<br />
shark.nethere.net- ldap<br />
ntdb-1.nethere.net<br />
winrestore64<br />
<br />
207.167.93.106- vmware6.eng 4156MB/8185MB 45.94GB/267GB free<br />
ntweb-2.nethere.net<br />
unixweb-12.nethere.net (no web sites on server)<br />
mailbox-1.nethere.net<br />
<br />
207.167.93.108- vmware8.eng<br />
mailx-2.nethere.net<br />
mx-2.nethere.net<br />
<br />
207.167.93.110- vmware10.eng<br />
unixweb-11.nethere.net<br />
</pre><br />
<br />
== Hardware Hosts ==<br />
<br />
<pre><br />
Cab 6-08<br />
<br />
siron-3 (off)<br />
siron-2<br />
siron-1<br />
106<br />
104<br />
sndg-br-1<br />
sndg-br-2<br />
<br />
mailbox-2.nethere.net (off - virtualized)<br />
mailbox-3.nethere.net <br />
mailbox-1.nethere.net (off - virtualized)<br />
<br />
scan-1 (OFF)<br />
<br />
scan-2 <br />
<br />
129.2<br />
102<br />
101<br />
<br />
105<br />
103<br />
<br />
Backup-2<br />
Backup-1<br />
<br />
</pre><br />
<br />
<pre><br />
Cab 5-02<br />
<br />
ntweb-3<br />
unixweb-7<br />
<br />
unixweb-3<br />
unixweb-2 (now virtualized)<br />
unixweb-10<br />
<br />
ntdb-2 or 3?<br />
<br />
vmware6.eng<br />
<br />
<br />
</pre><br />
<br />
== Admin Access ==<br />
<br />
Access to the Admin Systems are protected by firewalls and<br />
application filters based on IP addresses.<br />
<br />
=== Main Firewall ===<br />
The Main firewall is on sndg-cr-1. The main firewall is a Cisco 3750.<br />
<br />
==== IP Access List Editing ====<br />
<br />
<pre><br />
1. enable<br />
<br />
2. show ip access-lists access-list-name <br />
<br />
show ip access-list BLOCKED<br />
<br />
3. configure terminal<br />
<br />
4. ip access-list resequence access-list-name starting-sequence-number increment<br />
<br />
5. ip access-list {standard | extended} access-list-name<br />
<br />
ip access-list extended BLOCKED <br />
<br />
6. sequence-number permit source source-wildcard<br />
sequence-number deny protocol source source-wildcard destination destination-wildcard [precedence precedence] [tos tos] [log] [time-range time-range-name] [fragments]<br />
no sequence-number <br />
<br />
7. end<br />
<br />
8. show ip access-lists access-list-name<br />
</pre><br />
<br />
=== Host Firewall ===<br />
On the servers, there is often a firewall such as ipfw, ipf, or pf.<br />
<br />
==== Toolbox ====<br />
<br />
From root@admin-1 connect to scribe <br />
ssh scribe<br />
<br />
Stop the scrolling of messages to the screen<br />
/etc/rc.d/syslogd stop<br />
<br />
Add the new IP address to the pf config.<br />
vi /etc/pf.conf<br />
<br />
Reload the firewall rules<br />
pfctl -F all -f /etc/pf.conf<br />
<br />
Add the new IP address to the apache config.<br />
cd /nethere/conf/apache/conf/vhosts<br />
co -l prov.nethere.net scribe.nethere.net.common<br />
vi prov.nethere.net<br />
vi scribe.nethere.net.common<br />
ci -u prov.nethere.net scribe.nethere.net.common<br />
<br />
Reload the apache config<br />
apachectl restart<br />
<br />
Restart the syslog daemon<br />
/etc/rc.d/syslogd start<br />
<br />
==== Nagios ====<br />
<br />
From root@admin-1 connect to pike <br />
ssh nagios<br />
<br />
Add the new IP address to the apache config.<br />
cd /nethere/conf/apache/conf/vhosts<br />
co -l nagios.nethere.net cacti.nethere.net mrtg.nethere.net<br />
vi nagios.nethere.net cacti.nethere.net mrtg.nethere.net<br />
ci -u nagios.nethere.net cact.nethere.net mrtg.nethere.net<br />
<br />
Reload the apache config<br />
apachectl restart<br />
<br />
=== Application Filter ===<br />
In the applications, there is often a restriction in the host application<br />
configuration file.<br />
vi /usr/local/etc/apache/conf/vhosts/<host>.conf<br />
<br />
Restart the application<br />
apachect restart<br />
<br />
<br />
== NetHere/zNET/simplyweb Cert replacement ==<br />
<br />
<br />
If you wish to use the old CSR, Global Sign keeps the old<br />
CSR, so all you have to replace is the Cert.<br />
<br />
<li>To generate a new csr for each<br />
<pre><br />
openssl req -nodes -newkey rsa:2048 -keyout wild.nethere.key -out wild.nethere.csr<br />
openssl req -nodes -newkey rsa:2048 -keyout wild.simplyweb.key -out wild.simplyweb.csr<br />
openssl req -nodes -newkey rsa:2048 -keyout wild.znet.key -out wild.znet.csr<br />
</pre><br />
<br />
<li>Get a wild card cert for each company<br />
(nethere.net, simplyweb.net. znet.net)<br />
<br> <br />
https://www.globalsign.com/en/<br />
<br><br />
<br />
<li>Update certs, keys, and/or pem files on the following servers.<br />
<br />
<pre><br />
webmail-1.mail (webmail for nh, si, and zn) /nethere/conf/apache/pki/ssl.crt<br />
prov-1.mail (cp for nh, si, and zn) /nethere/conf/apache/pki/ssl.crt<br />
mta-1 and mta-2 (smtp) admin-2:/dist/files/nhmta/nethere/conf/postfix/nh/ssl<br />
/nethere/conf/postfix/nh/ssl<br />
/nethere/conf/postfix/si/ssl<br />
/nethere/conf/postfix/zn/ssl<br />
mailbox-1, mailbox-2, and mailbox-3 (pop3 and imap) (courier-imap)<br />
/nethere/conf/courier-imap/nh<br />
/nethere/conf/courier-imap/si<br />
/nethere/conf/courier-imap/zn<br />
(smtp)<br />
</pre><br />
<br />
</pre><br />
<br />
== Provisioning ==<br />
<br />
<br />
<br />
=== Webservers ===<br />
<br />
<br />
OS: FreeBSD 4.11-RELEASE-p10 #23<br />
<br />
<br />
==== General ====<br />
<br />
<pre><br />
unixweb-1 through 8<br />
<br />
All have the same configuration except for the following exceptions:<br />
<br />
unixweb-4: ZN Front Page server (depreciated)<br />
unixweb-5: ZN Front Page, Miva Merchant server<br />
unixweb-6: SI shared counter server (for SI sites): /www/lucy.inetworld.net/htdocs/cgi-bin/Count.cgi <br />
<br />
Server Software Installed<br />
<br />
Apache<br />
# httpd -v<br />
Server version: Apache/2.0.55<br />
Server built: Apr 5 2006 17:04:01<br />
PHP<br />
# /usr/local/bin/php4 -v<br />
PHP 4.4.2 (cgi-fcgi) (built: Mar 2 2006 09:31:57)<br />
<br />
Miva Merchant (unixweb-5)<br />
MySQL<br />
# mysqladmin version -u root -p<br />
Server version 4.0.25<br />
<br />
Apache configurations are in the following directories:<br />
<br />
/nethere/conf/apache/conf/nvhosts (name based hosting); naming convention is www.domain.tld<br />
/nethere/conf/apache/conf/vhosts (ip based,SSL); naming convention is www.domain.tld (IP information) and www.domain.tld.common (general site information)<br />
<br />
NetHere specific scripts are located in:<br />
/nethere/sbin<br />
<br />
Checking Server/site status via mod_status apache module:<br />
<br />
# apachectl stop<br />
# /usr/local/etc/rc.d/httpd.sh start-status<br />
<br />
URL: http://unixweb-#/status (replace # with the unixeweb number)<br />
** NOTE: After viewing status, do the following, otherwise other users can also view status by using .htaccess.<br />
# apachectl stop<br />
# apachectl startssl<br />
</pre><br />
<br />
==== Provisioning ====<br />
Provisioning new sites:<br />
<br />
*** All website provisioning is done via NH script (webadd) on sawfish (admin-1.nethere.net) ***<br />
# /nethere/sbin/webadd -h for usage<br />
<br />
Note: We've discontinued new website provisioning on all servers except for unixweb-6, unless otherwise noted for domain<br />
<br />
<br />
==== FTP Password Adjustment ====<br />
<br />
===== Unix Server =====<br />
To change an FTP password:<br />
<br />
On the customers server as root do<br />
<br />
passwd <username><br />
<br />
(The password prompt will not echo entry)<br />
<br />
===== Windows Server =====<br />
<br />
To change an FTP password:<br />
<br />
<pre><br />
Find the users login (ie aa5014) in the provisioning tool.<br />
Use Remote Desktop to get into the server.<br />
On Desktop find FTP application Icon.<br />
Use that tool to update the user's password.<br />
</pre><br />
<br />
==== Domain Aliasing ====<br />
<br />
For domain aliasing:<br />
<br />
1) Edit the /named/named.master on phoenix (ns1.nethere.net)<br />
a) Checkout the file; open with vi<br />
# co -l named.master<br />
# vi named.master<br />
b) find the domain that's serving as the master domain, then, following the general format of the file to add the domain aliases under the master domain entry, using the master domain zone file<br />
i.e. <br />
--cut--<br />
zone "domain.tld" { type master; file "master/domain.tld"; };<br />
zone "aliased_domain.tld" { type master; file "master/master_domain.tld"; }; <br />
--cut--<br />
c) save the file, then check-in:<br />
# ci -u named.master < /dev/null<br />
d) update the zone records, using the makefile in /named; check for errors<br />
# make new-zone<br />
# tail /var/log/named<br />
**Note: some domains are IP based virtually hosted (i.e. SSL certificate). This requires a modification to the standard aliasing procedure. If this is the case, do the following<br />
e) Create a new zone file called "domain.tld-alias" in /named/master, copy current domain.tld zone file to the domain.tld-alias file, and adjust the "website" records, removing whatever IP based information is there, and replacing with the appropriate $INCLUDE name-based host template.<br />
e.g.<br />
tropicalshade.net:<br />
--cut--<br />
;; ntweb-4<br />
@ IN A 66.63.136.4<br />
www IN A 66.63.136.4<br />
--cut--<br />
tropicalshade.net-alias<br />
--cut--<br />
;; website<br />
$INCLUDE master/nvhost.ntweb-4<br />
--cut--<br />
f) Adjust the named.master zone file in /named accordingly:<br />
i.e. <br />
--cut--<br />
zone "aliased_domain.tld" { type master; file "master/master_domain.tld-alias"; }; <br />
--cut--<br />
<br />
<br />
2) Edit the Apache configuration for the domain on the web server the master domain is hosted on<br />
a) Script is /nethere/sbin/webalias<br />
# /nethere/sbin/webalias -h (for usage)<br />
e.g. # /nethere/sbin/webalias -d www.originaldomain.tld -a www.domainalias.tld<br />
<br />
<br />
==== CGI Scripting ====<br />
<br />
For security, we've implemented suexec on all sites.<br />
All CGI *must* be placed in the cgi-bin/ (and/or cgibin/ directory for unixweb-4, 5) directory<br />
<br />
CGI permissions break down as follows:<br />
<br />
Ownership: user=username, group=webuser<br />
Permission: cgi-bin/ = 755; files = 755<br />
<br />
suexec errors can be found here:<br />
/www/default/logs/suexec_log<br />
<br />
cgi errors for a domain can be found in the main apache error logs for the domain:<br />
/www/www.domain.tld/logs/error_log<br />
<br />
More information on CGI scripting can be found here:<br />
http://httpd.apache.org/docs/1.3/howto/cgi.html<br />
<br />
<br />
==== Formmail Provisioning ====<br />
<br />
<br />
Formmail is provisioned via NH script (newformmail) on server domain is hosted on.<br />
# /nethere/sbin/newformmail -h for usage<br />
<br />
Usage: newformmail [-h] [-d domain] [-i ip_address] [-e "email1 email2"]<br />
<br />
<br />
==== Disabling a site ====<br />
<br />
<br />
To disable/enable a website, there is a script (webstatus) located on each webserver:<br />
/nethere/sbin/webstatus -h for usage<br />
<br />
1) Disable: <br />
# /nethere/sbin/webstatus -r -d -s www.domain.tld<br />
<br />
2) Enable:<br />
# /nethere/sbin/webstatus -r -e -s www.domain.tld<br />
<br />
==== Stats Provisioning ====<br />
<br />
We utilize webalizer for statistics.<br />
<br />
Stats are provisioned via NH scripts on server domain is hosted on.<br />
<br />
1) Provision stats for the domain (webaddstats_unix):<br />
# /nethere/sbin/webaddstats_unix -d www.domain.tld<br />
<br />
2) Run stats for the domain (runstats_unix):<br />
# /nethere/sbin/runstats_unix www.domain.tld<br />
<br />
More information on Webalizer can be found here:<br />
http://www.mrunix.net/webalizer<br />
README: ftp://ftp.mrunix.net/pub/webalizer/README<br />
<br />
<br />
==== Removing a site ====<br />
<br />
1) Site removal is primarily done via a NH script (webdel) on the webserver the domain is hosted on:<br />
# /nethere/sbin/webdel -h for usage<br />
# /nethere/sbin/webdel -r -d www.tldomain.com <br />
<br />
2) Update DNS accordingly (ns1.nethere.net)<br />
a) Checkout /named/named.master<br />
# co -l named.master<br />
b) Remove line containing domain<br />
c) Checkin /named/named.master<br />
# ci -u named.master < /dev/null<br />
d) Update zones<br />
# make new-zone<br />
e) Move zone files from /named/master to /named/archive<br />
# mv domain<br />
<br />
==== starter removal ==== <br />
<br />
All starter sites on home-1.nethere.net<br />
<br />
1) Removal primarily done via a NH script (webdel_home) on home-1<br />
# nethere/sbin/webdel_home -h for usage<br />
i.e. webdel_home -p zn -s startername <== for removing znet<br />
<br />
<br />
<br />
==== Disk Quota increases ====<br />
<br />
<br />
Quotas are increased by using the "edquota" command:<br />
# edquota username<br />
Note that quotas are in KB, so a conversion is necessary. Also note that the "soft" quota is 1 MB less than the "hard" quota.<br />
<br />
The calculations are as follows:<br />
"soft" = ( quota(in MB) - 1 ) x 1024<br />
"hard" = quota(in MB) x 1024<br />
e.g. for a 500 MB quota:<br />
soft => (500 - 1) x 1024 = 510976<br />
hard => 500 x 1024 = 512000<br />
<br />
To check a quota:<br />
# quota -v username<br />
<br />
You can also use the prototype users:<br />
<br />
# edquota -p quota100 username (100 MB)<br />
# edquota -p quota500 username (500 MB)<br />
# edquota -p quota1000 username (1000 MB)<br />
<br />
==== Granting of shell - Enterprise packages ====<br />
<br />
<br />
Shell is granted by the "chsh" command. By default, C shell (csh) is used:<br />
# chsh -s shell_needed username<br />
e.g. # chsh -s csh username<br />
<br />
*Note: customers must provide us with the static IP address(es) they will be connecting from. The IP address(es) must be added to the configuration on admin-1.<br />
1) Modify the corresponding rules files on admin-1:<br />
/dist/files/nhweb/etc/ipf.rules.fxp0 , .em0 (RCS controlled)<br />
2) Push updates to servers<br />
# cd /dist/rdist<br />
# gmake nhweb<br />
# gmake nhweb-update<br />
3) Reload the rules on the corresponding server that needs access to<br />
# ipf -Fa -f /etc/ipf.rules<br />
<br />
<br />
==== SSL certificate installs ====<br />
<br />
<br />
<br />
1) Whois the site for information on Registrant, email - to be used for SSL generation<br />
# whois domain.tld<br />
<br />
2) Create SSL self signed certificate, get CSR for customer to sign<br />
a) SSL generation is done by script: /nethere/conf/apache/pki/newsslcert.sh <br />
b) Copy CSR for customer<br />
<br />
3) Create/update Apache configuration files via NH script (webadd_ssl)<br />
# /nethere/sbin/webadd_ssl -h for usage<br />
a) webadd_ssl [-h] [-d domain] [-s ssl_domain] [-n]<br />
# webadd_ssl -d www.domain.tld -s www.domain.tld <br />
<br />
4) Update DNS zone with new IP address - done on ns1.nethere.net<br />
a) Check out DNS zone file in /named/master<br />
# co -l domain.tld<br />
b) Adjust A records:<br />
--cut--<br />
;; unixweb-## << enter the unixweb server number for ease of ID<br />
@ IN A vhost_ip_address<br />
www IN A vhost_ip_address<br />
--cut-- <br />
c) Check in DNS zone file<br />
# ci -u domain.tld < /dev/null<br />
d) Reload zone file<br />
# rndc reload domain.tld<br />
e) check /var/log/named for errors<br />
<br />
5) Update DNS PTR record for IP address - done on ns1.nethere.net<br />
Note: Assuming IP address a.b.c.d<br />
a) Check out in-addr.arpa zone for IP address in /named/master<br />
# co -l a.b.c <br />
b) Follow format for PTR records --cut--<br />
d IN PTR www.domain.tld.<br />
--cut--<br />
c) Check in PTR zone file # ci -u a.b.c < /dev/null<br />
d) Reload PTR zone <br />
# rndc reload c.b.a.in-addr.arpa<br />
e) check /var/log/named for errors<br />
<br />
<br />
==== Updating SSL certificate ====<br />
<br />
<br />
1) cd to /nethere/conf/apache/pki/ssl.crt on server site is hosted on<br />
<br />
2) Check out (RCS) www.domain.tld.crt file<br />
# co -l www.domain.tld.crt<br />
<br />
3) Edit the file, remove old certificate, paste in new certificate<br />
<br />
4) Check in (RCS) the www.domain.tld.crt file<br />
# ci -u www.domain.tld.crt < /dev/null<br />
<br />
5) Check, restart Apache<br />
# apachectl configtest<br />
# apachectl stop<br />
# apachectl startssl<br />
<br />
6) Verify httpd started: <br />
# ps auxw | grep httpd<br />
a) If no processes, will need to revert back to old SSL cert, and restart apache. Check logs for errors<br />
# view /www/default/logs/ssl_engine.log<br />
b) Check for "Unable to configure RSA server private key" and "key values mismatch" entries - this means a bad SSL certificate<br />
<br />
<br />
<br />
==== Password protection ====<br />
<br />
<br />
Http (simple) password protection is governed by the Apache configuration for the domain<br />
<br />
1) Create userdb, users file in the domain root directory (/www/www.domain.tld):<br />
# mkdir userdb<br />
# cd userdb<br />
# htpasswd -bc users username password<br />
<br />
2) Check out Apache config for www.domain.tld in /nethere/conf/apache/conf/<vhosts,nvhosts><br />
# co -l www.domain.tld(.common) <br />
<br />
3) Edit Apache configuration, add the following lines within the VirtualHost container <br />
--cut--<br />
<Directory "/www/www.domain.tld/dir_to_be_protected"><br />
AuthType Basic<br />
AuthName "www.domain.tld/dir_to_be_protected authentication"<br />
AuthUserFile /www/www.domain.tld/userdb/users <br />
<Limit GET POST><br />
require valid-user<br />
</Limit><br />
</Directory><br />
--cut--<br />
<br />
4) Check in Apache config<br />
# ci -u www.domain.tld(.common) < /dev/null <br />
<br />
5) Restart Apache<br />
# apachectl configtest<br />
# apachectl restart<br />
<br />
More on http (simple) password protection can be found here:<br />
http://httpd.apache.org/docs/1.3/howto/auth.html#basic<br />
<br />
<br />
==== .htaccess ====<br />
<br />
<br />
Used if customers want control of certain Apache directives (i.e. Authentication, etc.)<br />
<br />
1) Check out Apache config for www.domain.tld in /nethere/conf/apache/conf/<vhosts,nvhosts><br />
# co -l www.domain.tld(.common) <br />
<br />
2) Add the AllowOverride directive in the <Directory> section, under the PHP FCGIWrapper<br />
i.e.<br />
<Directory "/www/www.domain.tld/htdocs"><br />
FCGIWrapper /www/www.domain.tld/htdocs/cgi-bin/php4 .php<br />
AllowOverride AuthConfig FileInfo Indexes Limit<br />
</Directory><br />
<br />
3) Check in Apache config<br />
# ci -u www.domain.tld(.common) < /dev/null <br />
<br />
4) Restart Apache<br />
# apachectl configtest<br />
# apachectl restart<br />
<br />
<br />
More info on Allow Override can be found here: http://httpd.apache.org/docs/1.3/mod/core.html#allowoverride<br />
<br />
<br />
==== Domain re-provisioning ====<br />
<br />
<br />
## NOTE: Following is for domain being provisioned on same server (i.e. just being renamed)<br />
<br />
1) Adjust DNS - ns1.nethere.net<br />
a) Rename the DNS zone file to the new domain, remove the old DNS zone files from /named/master and /named/master/RCS<br />
b) Check in the new DNS zone file<br />
c) Edit /named/named.master - replace the old domain with the new one, alias as necessary<br />
d) Reload DNS zones - make new-zone in /named<br />
<br />
2) Adjust Apache config - server domain is hosted on<br />
a) Find current config file(s) in /nethere/conf/apache/conf/<vhosts,nvhosts><br />
b) Replace old domain name entries with new domain name, alias as necessary<br />
c) Save as new_domain.tld<br />
d) Remove old_domain.tld(.common), RCS/old_domain.tld(.common)<br />
e) Check in new_domain.tld via RCS:<br />
# ci -u new_domain.tld < /dev/null<br />
f) Check out Apache include configuration file via RCS /nethere/conf/apache/conf/<nvhosts.conf,vhosts.conf> <br />
g) Edit nvhosts.conf or vhosts.conf, replace old_domain.tld entries with new_domain.tld<br />
h) Check in via RCS /nethere/conf/apche/conf/<nvhosts.conf,vhosts.conf><br />
<br />
3) Rename directory for new domain<br />
# cd /www<br />
# mv www.old_domain.tld www.new_domain.tld<br />
<br />
4) Adjust PHP stub files/configuration<br />
a) Adjust php.ini file in /www/www.domain.tld/(php4,php5)<br />
b) Adjust PHP stub files in /www/www.domain.tld/htdocs/cgi-bin/(php4,php5) <br />
*note: need to chflags to "noschg" for /www/www.domain.tld/htdocs/cgi-bin/(php4,php5) before being able to update the stub files<br />
after adjusting, be sure to chflags schg /www/www.domain.tld/htdocs/cgi-bin/(php4,php5)<br />
<br />
5) Edit password file (vipw), replace old_domain.tld entries with new_domain.tld<br />
# vipw<br />
<br />
6) Restart Apache<br />
# apachectl configtest<br />
# apachectl restart<br />
<br />
## NOTE: Following is for domain being re-provisioned on new server<br />
<br />
1) Run NH script "webadd" on sawfish to provision domain on new server, *DO NOT* reload DNS, use same user/pass as before.<br />
2) After customer has uploaded site to new server and gives the ok do the following:<br />
a) Update DNS zone for domain.tld; reload zone for domain.tld<br />
b) Wait 48 hours, then remove the site off the old server<br />
# /nethere/sbin/webdel<br />
<br />
==== FTP space provisioning ====<br />
<br />
<br />
Generally FTP sites are provisioned on the server that hosts the main website. In the case of NT based FTP sites, we usually provision them on the server with the most space available.<br />
<br />
1) Provision site on server<br />
a) Done via NH script (webadd_ftp): webadd_ftp [-h] [-d domain] [-u username] [-p password] [-n]<br />
# /nethere/sbin/webadd_ftp -h <- for usage<br />
EX: for ftp.domain.tld:<br />
# /nethere/sbin/webadd_ftp -d ftp.domain.tld -u username -p password<br />
(leaving off the -n will restart the proftpd process)<br />
b) Note: The host IP address will be given when the provisioning is completed, use that ip (a.b.c.d) for DNS entries<br />
c) Note: If this is an existing customer on the server, you'll need to increase the quota manually by 100 MB for the customer, see the section on quota increases for more info.<br />
<br />
2) Adjust DNS for domain on ns1.nethere.net<br />
a) Checkout zone for domain<br />
b) Add ftp host entry for domain, adjust serial<br />
EX for domain.tld:<br />
--cut--<br />
ftp IN A a.b.c.d<br />
--cut--<br />
c) Check in zone for domain<br />
d) Reload zone<br />
# rndc reload domain.tld<br />
3) Adjust PTR record for domain, adjust serial<br />
a) Checkout zone for a.b.c<br />
b) Add record for domnain:<br />
--cut--<br />
d IN PTR ftp.domain.tld.<br />
--cut--<br />
c) Check in zone for a.b.c<br />
d) Reload zone<br />
# rndc reload c.b.a.in-addr.arpa<br />
4) Check for DNS errors<br />
a) tail /var/log/namedb<br />
<br />
<br />
<br />
==== SiteBuilder provisioning ====<br />
<br />
<br />
unixweb-7.nethere.net<br />
<br />
*Must re-provision site on unixweb-7.nethere.net (if not already done)<br />
<br />
1) Log in to SB admin:<br />
http://sitebuilder.nethere.net/admin<br />
a) username: root<br />
<br />
2) Add site to SB config<br />
a) Click on Site Management -> Add regular<br />
* Alias is website username: i.e. aa####<br />
* Check the "Active" box <br />
* Plan is "BasePlan" <br />
* Password same as website<br />
b) Click on "Publish Properties<br />
* Check "Allow publishing"<br />
* Site host name: www.domain.tld<br />
* FTP host: unixweb-7.nethere.net<br />
* FTP login/password: same as site user/pass<br />
* FTP working directory: leave blank<br />
c) Click Apply<br />
<br />
<br />
<br />
<br />
==== Name Servers ====<br />
<br />
<br />
OS: FreeBSD<br />
<br />
<br />
==== General ====<br />
<br />
<br />
ns1.nethere.net - Primary name servers for DNS zone records<br />
/named - contains the files that have all DNS domain zone entries (named.master, named.slave, named.acl) plus Makefile for distributing DNS records<br />
/named/master - contains all the domain zone files for which we are authorative for, as well as IP address (PTR records)<br />
<br />
ns2.nethere.net - Secondary (slave) name server for DNS zone records<br />
<br />
nsrbl-1.nethere.net - RBL (Realtime Blackhole List) DNS server<br />
/named/rbldns/cache - contains the files for domains that we specifically allow or deny<br />
<br />
nscache-1,2 - caching name servers<br />
<br />
<br />
<br />
==== Adding DNS website entries ====<br />
<br />
<br />
1) Create a DNS zone file for domain.tld via NH script (zoneadd_vhost)<br />
# /nethere/sbin/zoneadd_vhost -h (for usage)<br />
<br />
<br />
<br />
==== Adding DNS IP entries ====<br />
<br />
<br />
<br />
1) Create the forward and reverse DNS records via NH script (zone_generate) for a netblock, will create /tmp/customer.forward and /tmp/customer.reverse files to be read<br />
a) /nethere/sbin/zone_generate -h for usage<br />
i.e. for netblock a.b.c, starting IP d, ending IP z<br />
# zone_generate -n a.b.c -b d -e z -p customer<br />
<br />
2) Checkout, edit the IP in-addr.arpa zone file, reload the zone<br />
a) # co -l a.b.c<br />
b) Seach for the nearest netblock area for the domain, follow format for customer info, read in the /tmp/abbrev.rdns file accordingly, increase Serial for zone in YYYYMMDD## format<br />
e.g.<br />
--cut--<br />
;;;;<br />
;; 66.63.152.232/30 (255.255.255.252)<br />
;; Description: First Choice Home Improvement<br />
;; Contact: Shannon Hill <firstchoicehi@hotmail.com>, (858) 277-5351<br />
;; Location: AR-1, Serial3/0/18:0<br />
;;;;<br />
232 IN PTR firstchoice-net.access.nethere.net.<br />
233 IN PTR firstchoice-gw.access.nethere.net.<br />
234 IN PTR firstchoice-2.access.nethere.net.<br />
235 IN PTR firstchoice-bcast.access.nethere.net.<br />
--cut--<br />
c) # ci -u a.b.c < /dev/null<br />
d) # rndc reload c.b.a.in-addr.arpa<br />
e) verify loading of zone: # tail /var/log/named<br />
3) Checkout, edit the forward DNS zone file, reload the zone<br />
a) # co -l access.nethere.net<br />
b) Search for the nearest neblock area for the domain, follow format for customer, read in the /tmp/abbrev.fdns file accordingly, increase Serial for zone in YYYYMMDD## format<br />
e.g.<br />
--cut--<br />
;; 66.63.152.232/30 (255.255.255.252)<br />
firstchoice-net IN A 66.63.152.232<br />
firstchoice-gw IN A 66.63.152.233<br />
firstchoice-2 IN A 66.63.152.234<br />
firstchoice-bcast IN A 66.63.152.235<br />
--cut--<br />
c) # ci -u access.nethere.net < /dev/null<br />
d) # rndc reload access.nethere.net <br />
e) verify loading of zone: # tail /var/log/named<br />
<br />
<br />
<br />
<br />
==== Unblocking RBLd IP addresses ====<br />
<br />
There are two primary reasons why we add customers to the allow relay list:<br />
a) They have a static IP address (i.e. DSL, T1) and wish to use our mail servers<br />
b) They've been blocked by one of our subscribed blackhole lists, however, have patched their machine and are no longer open to relay.<br />
<br />
To do this on nsrbl-1.nethere.net:<br />
<br />
1) Checkout the allow.relays.nethere.net file located in /named/rbldns/cache<br />
<br />
2) Edit the file, and add the IP address in the following format:<br />
--cut--<br />
a.b.c.d YYYYMMDD hostname reason for listing<br />
--cut--<br />
<br />
You can also add subnets via '/' notation for relay<br />
--cut--<br />
a.b.c.d/28 YYYYMMDD hostname reason for listing<br />
--cut--<br />
<br />
3) Check in the file<br />
<br />
4) Updates to the rbldns zone are done automatically on the hour, every hour, so no need to do anything else. <br />
<br />
Note:<br />
<br />
For "permanent" (i.e. customer static IP addresses), add the IP address in the "## permanent allowed relay (i.e. customer w/ static IP)" section<br />
For "temporary" (i.e. blackhole listed IP addresses), add the IP address in the "## temporary" section<br />
<br />
Also, for temporary IP addresses, we need to send the note to the requester detailing the following:<br />
a) If the IP address relays Spam/UCE/Viruses through us, it is to be removed permanently.<br />
b) The customer must follow the steps listed on the blackhole list the IP was listed on to get removed.<br />
<br />
<br />
<br />
==== Flushing DNS cache for a domain ====<br />
<br />
<br />
Due usually to a bad zone or excessively long TTL for a domain, the cache for it will need to be flushed.<br />
<br />
On nscache-1:<br />
# /usr/local/sbin/rndc flushname domain.tld<br />
<br />
If cache is still corrupted (i.e. zone lookups either fail or are incorrect), need to stop and restart the caching server:<br />
<br />
# /etc/init.d/local.named stop<br />
# /etc/init.d/local.named start<br />
<br />
==== SiteBuilder ====<br />
http://sitebuilder.nethere.net/<br />
-URL used to test sitebuilder<br />
<br />
http://sitebuilder.nethere.net/Login<br />
-URL for control panel. <br />
<p>This is where the customer also logs in to manage<br />
their web site and also where you log in to administer sitebuilder.<br />
Sitebuilder is hosted on sb-2.nethere.net. The site is designed/built on<br />
this server and published to unixweb-7.nethere.net. This is the only server<br />
that can host a sitebuilder web site. The admin log in to manage<br />
sitebuilder is:<br />
<br />
Username: admin<br />
Password: N3tH3r31!<br />
<br />
==== Cart32 ====<br />
cart32 information:<br />
<br />
Cart32 is hosted on ntweb-6.nethere.net. That is where all of the<br />
configuration files are located. The location of the ini file is<br />
D:\websites\Cart32cgi/cart32.ini. This is where ip restrictions to admin<br />
panel are set, password can be reset for admin, time limit restriction<br />
reset, etc.<br />
<br />
There are four customers that still use cart32:<br />
<br />
<pre><br />
https://www.cart.simplyweb.net/lab400/cart/c32web.exe<br />
https://www.cart.simplyweb.net/nutragenics/cart/c32web.exe<br />
https://www.cart.simplyweb.net/retrogen/cart/c32web.exe<br />
https://www.cart.simplyweb.net/stonesculptorssupplies/cart/c32web.exe<br />
</pre><br />
<br />
The client codes are lab400, nutragenics, retrogen, and<br />
stonesculptorssupplies. You can reset their passwords through the admin<br />
panel.<br />
<br />
To administer cart32:<br />
<br />
Control Panel: https://www.cart.simplyweb.net/cart/c32web.exe/Admin<br />
Username: administrator<br />
Password: N3tH3r31!<br />
Cart Admin Password: N3tH3r31!<br />
<br />
=== Webmail ===<br />
<br />
<br />
<br />
==== Removing webmail filters ====<br />
<br />
All webmail filters are stored in the MySQL database on webmail-1.<br />
<br />
Simple SQL commands:<br />
<br />
* Choosing a DB<br />
mysql> use db_name;<br />
<br />
* Showing tables in a DB (must be using a DB)<br />
mysql> show tables;<br />
<br />
* Describing a table's fields<br />
mysql> describe table_name;<br />
<br />
<br />
For NetHere, the database is horde_nh.<br />
For Simply, the database is horde_si.<br />
For zNET, the database is horde_zn.<br />
<br />
1) Log to MySQL as root, using the normal root password<br />
a) # mysql -u root -p<br />
2) Select the appropriate database<br />
a) mysql> use horde_nh<br />
3) Delete webmail filters.<br />
a) mysql> delete from horde_prefs where pref_uid='<username>' and pref_name='filters';<br />
Replace <username> with the users login.<br />
<br />
<br />
==== Removing Address book entries ====<br />
<br />
1) Log in<br />
2) Select appropriate db<br />
3) Find object_id, delete<br />
a) mysql> select object_id from turba_objects where object_email='email@domain.tld';<br />
b) mysql> delete from turba_objects where object_id='object_id_obtained_previously';<br />
<br />
<br />
==== Repairing webmail address book ====<br />
<br />
<br />
1) Export (via webmail, if possible) the address book to a csv file.<br />
<br />
2) Delete the Turba objects in the mySQL database in the turba_objects table that is assigned to the customer.<br />
mysql> delete from turba_objects where owner_id='username';<br />
<br />
3) Import the saved address book via webmail.<br />
<br />
<br />
==== Changing webmail From Information ====<br />
<br />
1) Login as the customer (http://webmail.nethere.net)<br />
<br />
2) Click on Options<br />
<br />
3) Click on Personal Information<br />
<br />
4) Select either default identity or a new one and click on Edit Your Identities<br />
<br />
=== MySQL ===<br />
<br />
<br />
All unixweb-## boxes have MySQL processes running on them. DB provisioning is done on the same server as which hosts the domain.<br />
For Windows 2000 MySQL provisioning, all DBs are placed on ntdb-2.nethere.net<br />
<br />
<br />
==== MySQL DB provisioning ====<br />
<br />
<br />
Use https://phpmyadmin.nethere.net <br />
User: root<br />
PW: <root db pw><br />
<br />
1) Select the server that the DB needs to be hosted on, typically the same server that the website is hosted on.<br />
2) Create the DB<br />
a) Under "MySQL" - "Create new database" - the DB name is typically the website username; click "Create"<br />
3) Create the User, set permissions for user on DB<br />
a) Click "Home" (upper left), then click "Privileges" (Under "MySQL")<br />
b) Click "Add a new User" make sure "Any host" selected, "User name" is typically the website username, "Password" is typically the website password. !LEAVE "Global Privileges" settings deselected! Click "Go" (bottom right corner) when finished.<br />
c) Under "Database-specific privileges" be sure to add the user database<br />
d) Select DB, then for "Privileges:" make sure the following are checked:<br />
Select, Insert, Update, Delete, Create, Alter, Index, Drop, Create Temporary Tables, Lock Tables, References<br />
<br />
<br />
=== List Server ===<br />
<br />
lists-1.nethere.net<br />
<br />
<br />
==== Mail List provisioning ====<br />
<br />
<br />
1. Go to the /mailman/bin directory:<br />
# cd /mailman/bin<br />
<br />
2. Create the list on lists-1:<br />
# ./newlist -q \<br />
listname@lists.domain.com \<br />
mailman-owner@lists.domain.com \<br />
password <br />
<br />
3. Configure the list using default settings:<br />
# ./config_list -i /mailman/data/defaultlist.cfg listname<br />
<br />
*** IF A NEW DOMAIN FOLLOW INSTRUCTIONS BELOW ***<br />
<br />
NOTE: All files in /etc/mail are controlled by RCS<br />
<br />
4. Go to the /etc/mail directory:<br />
# cd /etc/mail<br />
<br />
5. Add the list domain to the mailertable file:<br />
lists.domain.com mailman:lists.domain.com<br />
<br />
6. Add the list domain to the relay-domains file.<br />
<br />
7. Add the list domain to the virtuserdomain file.<br />
<br />
8. Add the following entries to the virtusertable file:<br />
mailman@lists.domain.com <customer_email_address><br />
mailman-owner@lists.domain.com mailman@lists.domain.com<br />
<br />
9. Run make to rebuild all files:<br />
# make<br />
<br />
10. Run make to restart sendmail:<br />
# make restart<br />
<br />
<br />
=== Backup Server ===<br />
<br />
backup-1 -> odd numberd unix<br />
backup-2 -> even numbered unix<br />
<br />
<br />
On Saturdays, we do a level 0 dump<br />
the rest are differential<br />
<br />
Restoration fees are basically $25 per day restored for email, $150 for website from cancelled archive, $25 a day for website <br />
<br />
Restoration is based on customer request. In order to restore a site/mailbox fully, you must start restoration from the previous level 0 backup and continue until the day after the day requested, since backups are performed in the morning.<br />
<br />
General rule is to leave the gzipped file available for 24 hours, after that remove it.<br />
<br />
<br />
==== Site Restoral ====<br />
<br />
<br />
If the site was removed by the web_del script, then so long as it is within 30 days since removal, the gzipped site will be available within /www/archive/www.domain.tld-date_removed.tar.gz<br />
Extract with tar:<br />
# tar -zxvpf www.domain.tld-date_removed.tar.gz<br />
<br />
If the site needs to be restored from backup, here are the steps<br />
<br />
1) To determine where backups are stored, go to /nethere/conf/backup<br />
<br />
2) grep for the server within the directory to determine the backup directory used<br />
i.e. <br />
# grep unixweb-1 *<br />
<br />
4) Go to /backup1/unix/unixweb-1.nethere.net<br />
a) you'll see a bunch of directories, named <date>-<dump_level><br />
<br />
5) Since the backups happen in the morning, typically you'll need to go to the day before, restore<br />
e.g. for 11/10<br />
a) cd 20041109-3<br />
b) restore -if and whatever the filesystem is named<br />
(for interactive: add files/dirs, extract, then use 1 for volume #)<br />
i.e.<br />
restore> add <path_to_dir/files><br />
restore> extract<br />
(for volume #: 1)<br />
(Set owner: y)<br />
<br />
6) After you've restored the files, just use scp as root on sawfish to copy them off of backup-# and then onto unixweb-#<br />
a) (on sawfish - two step process)<br />
# scp backup-1:/tmp/<restored_file> /tmp<br />
# scp /tmp/<restored_files> unixweb-1:/tmp<br />
<br />
7) Cleanup any restored files off of backup-# server and sawfish<br />
<br />
<br />
With the newly restored files, following are the steps necessary to restore a site:<br />
<br />
1) Restore www.domain.tld directory to /www<br />
# mv /www/archive/www.domain.tld /www<br />
<br />
2) Restore configuration files<br />
a) Apache: mv /www/archive/nethere/conf/apache/(n)vhosts/www.domain.tld /nethere/conf/apache/conf/(n)vhosts/<br />
b) Webalizer: mv /www/archive/nethere/conf/webalizer/unix/www.domain.tld /nethere/conf/webalizer/unix<br />
<br />
3) Add user to /etc/master.passwd:<br />
a) vipw<br />
(go to end of file, read in the master.passwd file from /www/www.domain.tld)<br />
:r /www/www.domain.tld/master.passwd<br />
b) Remove /www/www.domain.tld/master.passwd<br />
<br />
4) Change flags on cgi-bin/ directories:<br />
# chflags sunlnk /www/www.domain.tld/htdocs/cgi-bin/<br />
# chflags schg /www/www.domain.tld/htdocs/cgi-bin/php*<br />
<br />
5) Add configuration file entry to nvhosts/vhosts.conf<br />
# co -l (n)vhosts.conf<br />
# vi (n)vhosts.conf<br />
# ci -u (n)vhosts.conf<br />
<br />
6) Check apache, restart; verify apache<br />
# apachectl configtest<br />
# apachectl graceful<br />
# <br />
<br />
7) Cleanup remaining restored files<br />
# rm -r /www/archive/www<br />
# rm -r /www/archive/nethere<br />
<br />
<br />
8) Enter zone into DNS on ns1.nethere.net<br />
a) zone file<br />
# mv /named/archive/domain.tld /named/master<br />
# mv /named/archive/domain.tld,v /named/master/RCS<br />
b) named.master file <br />
# co -l /named/named.master<br />
(add zone)<br />
# ci -u /named/named.master<br />
# cd /named; make new-zone<br />
# tail /var/log/named (look for errors)<br />
*************************************************<br />
(12:40:47 PM) Henry Chan: restore is now available that is compatible with the 4.4bsd format<br />
(12:41:04 PM) Henry Chan: to restore, use the following command: restore -c -i -f path_to_archive<br />
(12:41:08 PM) Henry Chan: the "-c" is what does it<br />
(12:41:29 PM) Henry Chan: (only applies to backup-1-new... doesn't work on backup-2 or backup-4)<br />
<br />
<br />
==== Email Restoral ====<br />
<pre><br />
Storage path prefixes:<br />
/nfs/1 is sndg-netapp-1 (on backup-4 - /backup/hosts/2/e0.sndg-netapp-1.nethere.net)<br />
/nfs/2 is sndg-netapp-2 (on backup-3 - /backup/hosts/2/e0.sndg-netapp-2.nethere.net)<br />
/nfs/3 is sndg-netapp-3 (on backup-2 - /backup/hosts/2/e0.sndg-netapp-3.nethere.net)<br />
/nfs/4 is sndg-netapp-1 (on backup-2 - /backup/hosts/2/e0.sndg-netapp-1.nethere.net)<br />
/nfs/5 is sndg-netapp-2 (on backup-1 - /backup/hosts/2/e0.sndg-netapp-2.nethere.net)<br />
/nfs/6 is sndg-netapp-3 (on backup-2 - /backup/hosts/2/e0.sndg-netapp-3.nethere.net)<br />
<br />
Looking up a mailbox's Storage Path:<br />
You will need to know which directory and NFS server the customer's mail is <br />
stored on. Use the Provisioning Tool to get the Storage Path (find the customer's mail<br />
account, then click on the Engineering sub-tab under the Email tab). You should end up<br />
with something like "/nfs/3/nh/h/t/htchan/Maildir". Match the prefix of the storage<br />
directory with an NFS server (see prefixes above).<br />
<br />
<br />
<br />
1) If restoring a recently deleted mail account:<br />
a) Look for the archive on mailbox-1 in /nfs/archive/mail/{platform}/{username}.{date}-{PID}.tar.gz. <br />
If it doesn't exist, it's been too long and the only way to get email back is to restore from backup.<br />
b) Make sure the account is re-created in Provisioning Tool and look up the storage path <br />
(see "Looking up a mailbox's Storage Path" above).<br />
c) Extract the archive to a temporary directory:<br />
Sample command:<br />
cd /tmp; tar xzvpf /nfs/archive/mail/nh/zella.20090902-7508.tar.gz<br />
Sample output:<br />
nfs/2/nh/z/e/zella/<br />
nfs/2/nh/z/e/zella/Maildir/<br />
nfs/2/nh/z/e/zella/Maildir/tmp/<br />
nfs/2/nh/z/e/zella/Maildir/new/<br />
nfs/2/nh/z/e/zella/Maildir/cur/<br />
nfs/2/nh/z/e/zella/Maildir/maildirsize<br />
nfs/2/nh/z/e/zella/Maildir/.Trash/<br />
nfs/2/nh/z/e/zella/Maildir/.Trash/tmp/<br />
nfs/2/nh/z/e/zella/Maildir/.Trash/new/<br />
nfs/2/nh/z/e/zella/Maildir/.Trash/cur/<br />
nfs/2/nh/z/e/zella/Maildir/.Trash/maildirfolder<br />
nfs/2/nh/z/e/zella/Maildir/.Drafts/<br />
nfs/2/nh/z/e/zella/Maildir/.Drafts/tmp/<br />
nfs/2/nh/z/e/zella/Maildir/.Drafts/new/<br />
nfs/2/nh/z/e/zella/Maildir/.Drafts/cur/<br />
nfs/2/nh/z/e/zella/Maildir/.Drafts/maildirfolder<br />
nfs/2/nh/z/e/zella/Maildir/.Sent Items/<br />
nfs/2/nh/z/e/zella/Maildir/.Sent Items/tmp/<br />
nfs/2/nh/z/e/zella/Maildir/.Sent Items/new/<br />
nfs/2/nh/z/e/zella/Maildir/.Sent Items/cur/<br />
nfs/2/nh/z/e/zella/Maildir/.Sent Items/maildirfolder<br />
nfs/2/nh/z/e/zella/Maildir/courierpop3dsizelist<br />
Note:<br />
In the above output, notice that the Maildir is located in "nfs/2/nh/z/e/zella", it will be used in the next step.<br />
d) Copy the Maildir files to the new mailbox storage path using tar (tar does a better job at preserving <br />
things like symbolic links and permissions than cp or mv):<br />
Sample command:<br />
cd /tmp/nfs/2/nh/z/e/zella; tar cf - Maildir | (cd /nfs/3/nh/z/e/zella; tar xvpf -)<br />
Output:<br />
You should list the files as it is copied. Basically, similar to what you <br />
saw in step "c" above but without the "nfs/2/nh/z/e/zella" prefix.<br />
Note:<br />
The command format is basically:<br />
cd /tmp/{restored-maildir-path-see-note-in-step-c}; tar cf - Maildir | (cd {new-storage-path-without-Maildir}; tar xvpf -)<br />
e) Clean up temporary directory:<br />
Sample command:<br />
cd /tmp; rm -rf nfs<br />
<br />
2) If restoring a zfs mailbox (from the last week) to a particular date's backup:<br />
a) login to the correct sndg-netapp-[1/2/3]-new<br />
b) get the files from the correct zfs snapshot<br />
Sample commands:<br />
cd /tank0/mail/.zfs/snapshot/20180215-0/nh/c/h/christineat<br />
ls -l<br />
cd to the desired directory and copy the files over to the users mailbox<br />
cp -p /tank0/mail/.zfs/20180215-0/nh/c/h/christineat/Maildir/cur /tank0/mail/nh/c/h/christineat/Maildir/cur<br />
<br />
3) If restoring a mailbox to a particular date's backup:<br />
a) Look up the storage path to the customer's mailbox and determine which backup server the <br />
dump file is on (see "Looking up a mailbox's Storage Path" and "Storage path prefixes" above).<br />
b) Restore the level 0 and any incremental backups (in order) to /tmp on the backup server:<br />
Sample commands:<br />
cd /tmp<br />
restore4x -if /backup/hosts/2/e0.sndg-netapp-2.nethere.net/20090912-0/mail.dump<br />
chflags -R 0 /tmp<br />
restore4x -if /backup/hosts/2/e0.sndg-netapp-2.nethere.net/20090913-1/mail.dump<br />
chflags -R 0 /tmp<br />
...skipped repetitive stuff here...<br />
restore4x -if /backup/hosts/2/e0.sndg-netapp-2.nethere.net/20090918-6/mail.dump<br />
chflags -R 0 /tmp<br />
Note:<br />
After each restore, we need to recursively remove all flags from /tmp to eliminate <br />
the immutable flag that gets set on the files (this happens only on NetApp dumps... who knows why).<br />
c) Create a new tar of the Maildir directory so that it can be copied to mailbox-1 for further processing:<br />
Sample command:<br />
cd /tmp/nh/z/e/zella; tar cf /tmp/archive.tar Maildir<br />
d) Clean up /tmp:<br />
Sample command:<br />
rm -rf /tmp/nh<br />
Note:<br />
Depending on the platform, it might be /tmp/nh, /tmp/si, or /tmp/zn.<br />
e) Copy the /tmp/archive.tar file to /tmp on mailbox-1.<br />
f) On mailbox-1, extract the Maildir archive on top of what they already have:<br />
Sample command:<br />
cd /nfs/2/nh/z/e/zella; tar xvpf /tmp/archive.tar<br />
g) Remove /tmp/archive.tar.<br />
</pre><br />
<br />
==== Beginning of old instructions ====<br />
Use the same techniques as site restoration, with the following exceptions:<br />
<br />
1) Since everything is in maildir format, you have to reassmble things in /tmp/<username>, tar it up, and then restore it on any of the mailbox machines<br />
a) using the following syntax: <br />
# tar -zcpf /tmp/username.tgz yyyymmdd-#/platform yyyymmdd-#/platform yyyymmdd-#/platform<br />
i.e.<br />
# tar -zcpf /tmp/username.tgz 20050915-5/nh 20050914-4/nh 20050910-0/nh<br />
<br />
2) In regards to mail spools, you'll need to use the chflags command to adjust the flags on the files. For whatever reason, they are stored with a system immutable flag<br />
a) Do chflags -R noschg <dir> on the restored directory before copying the files with sawfish otherwise, the files cannot be deleted:<br />
i.e. <br />
# chflags -R noschg /backup1/unix/sndg-netapp-1-e2b.nethere.net/20050105-4/<br />
b) After tar/gzipping, remove the restored directories<br />
i.e<br />
# rm -r /backup1/unix/sndg-netapp-1-e2b.nethere.net/20050105-4/nh<br />
<br />
3) Can restore quickly with script on mailbox-1 (or mailbox-2) using the NH script "restoremail"<br />
a) # /nethere/sbin/restoremail -h for usage<br />
i.e. <br />
# restoremail -p nh -f nhusername.tgz -n 2 -u nhusername<br />
*** End of old instructions ***<br />
<br />
<br />
<br />
* Backup locations *<br />
<br />
backup-1:<br />
backup-3.nethere.net<br />
fpweb-1.nethere.net<br />
home-1.nethere.net<br />
koi.nethere.net<br />
marmaduke.inetworld.net<br />
ntdb-1.nethere.net<br />
ntweb-1.nethere.net<br />
ntweb-3.nethere.net<br />
ntweb-5.nethere.net<br />
ntweb-7.nethere.net<br />
phoenix.nethere.net<br />
unixweb-1.nethere.net<br />
unixweb-3.nethere.net<br />
unixweb-5.nethere.net<br />
unixweb-7.nethere.net<br />
shark.nethere.net<br />
tetra.nethere.net<br />
wms-1.nethere.net<br />
<br />
backup-2:<br />
andromeda.nethere.net<br />
backup-4.nethere.net<br />
eel.nethere.net<br />
fpweb-2.nethere.net<br />
ntdb-2.nethere.net<br />
ntweb-2.nethere.net<br />
ntweb-4.nethere.net<br />
ntweb-6.nethere.net<br />
unixweb-2.nethere.net<br />
unixweb-4.nethere.net<br />
unixweb-6.nethere.net<br />
ds.znet.com<br />
mx1.znet.com<br />
mx2.znet.com<br />
mx3.znet.com<br />
dmx.znet.com<br />
la.znet.com<br />
uf.znet.com<br />
<br />
backup-3:<br />
backup-1.nethere.net<br />
lists-1.nethere.net<br />
mailbox-1.mail.nethere.net<br />
mailbox-3.mail.nethere.net<br />
mta-1.mail.nethere.net<br />
mx-1.nethere.net<br />
nscache-1.nethere.net<br />
nsrbl-1.nethere.net<br />
pegasus.nethere.net<br />
scan-1.mail.nethere.net<br />
sndg-netapp-2.nethere.net<br />
relay-1.mail.nethere.net<br />
webmail-1.mail.nethere.net<br />
<br />
backup-4:<br />
ahi.nethere.net<br />
backup-2.nethere.net<br />
cp-1.nethere.net<br />
cygnus.nethere.net<br />
dragon.nethere.net<br />
lisa.nethere.net<br />
mailbox-2.mail.nethere.net<br />
mta-2.mail.nethere.net<br />
news-1.nethere.net<br />
nscache-2.nethere.net<br />
nsrbl-2.nethere.net<br />
sawfish.nethere.net<br />
scan-2.mail.nethere.net<br />
scribe.nethere.net<br />
sndg-netapp-1.nethere.net<br />
relay-2.mail.nethere.net<br />
urchin.nethere.net <br />
<br />
<br />
<br />
=== Mail servers ===<br />
==== Organization ====<br />
<pre><br />
<br />
<br />
NetHere Mail Servers<br />
====================<br />
<br />
<br />
<br />
<br />
Customers Inbound Mail Outbound Mail<br />
/\ || || /\<br />
|| || || smtp ||<br />
|| || \/ ||<br />
|| || ServerIron ||<br />
|| || || ||<br />
|| || || ||<br />
|| || smtp \/ ||<br />
|| || =========> mta-1 mta-2 ====> relay-1 <==== unixweb-* servers<br />
|| || || /\ relay-2 ntweb servers<br />
http || pop3 || || ||<br />
|| imap || \/ +--------------+<br />
|| || ServerIron |<br />
|| || || |<br />
|| || || |<br />
|| || \/ |<br />
|| || nsrbl-1 nsrbl-2 |<br />
|| || (spamhaus, dcc blacklist) |<br />
|| || || |<br />
|| \/ \/ |<br />
\/ mailbox-1 <=== scan-1 scan-2 |<br />
webmail <===> mailbox-2 (sendmail/amavisd/sophos) |<br />
imap mailbox-3 |<br />
/\ /\ |<br />
|| || \/<br />
nfs || +---------------------------> mta-db<br />
|| (LDAP)<br />
\/<br />
sndg-netapp-1<br />
sndg-netapp-2<br />
sndg-netapp-3<br />
<br />
<br />
</pre><br />
<br />
==== Overview ====<br />
mta-1.mail.nethere.net- Primary servers for all inbound and outbound mail.<br />
mta-1.mail.nethere.net- Primary servers for all inbound and outbound mail.<br />
<br />
nsrbl-1.nethere.net- Realtime blacklist check using dccd and spamhaus. The mta servers query the ip address of the mail server against the blacklist before transferring the email to the scanning servers.<br />
<br />
nsrbl-2.nethere.net<br />
<br />
scan-2.mail.nethere.net- Scans email with Sophos (currently disabled) and Spam Assasin before sending it to the mailbox servers.<br />
scan-2.mail.nethere.net Scans email with Sophos (currently disabled) and Spam Assasin before sending it to the mailbox servers.<br />
<br />
mailbox-1.mail.nethere.net- Front end server for the mail directories. Queries LDAP server for miscellaneous customer information including the directory the email should be stored in, passwords, spam sensitivity levels, etc. before delivering email to appropriate directory.<br />
mailbox-2.mail.nethere.net<br />
mailbox-3.mail.nethere.net<br />
<br />
Mail is stored in maildir format:<br />
<br />
/nfs/<nfs_number>/<platform>/u/s/username<br />
<br />
sndg-netapp-1-new.nethere.net- Solaris system using zfs set up as a nfs mount on mailbox-1, -2, and -3. Email is stored on these platforms for all customers.<br />
sndg-netapp-2-new.nethere.net<br />
sndg-netapp-3-new.nethere.net<br />
<br />
mx-1.nethere.net- spooling mail server<br />
<br />
mailx-1.nethere.net- all outbound mails originating from a web server is sent out this email server. All web servers are set to use mailx.nethere.net which resolves (through the serveriron) to either mailx-1.nethere.net and mailx-2.nethere.net. Since we shut down mailx-2, all outbound mail from the web servers goes through mailx-1. This was to prevent all the customers email from being blacklisted as a result of a web site being compromised.<br />
<br />
mta-db.mail.nethere.net- database server for the mta servers. All postfix databases are stored here.<br />
<br />
webmail.nethere.net- Hosts the webmail program for webmail.nethere.net, webmail.simplyweb.net and webmail.znet.net.<br />
<br />
==== Client mail settings ====<br />
<br />
Inbound Mail Server:<br />
<br />
POP3<br />
pop3.nethere.net<br />
pop3.znet.net<br />
pop3.simplyweb.net<br />
<br />
IMAP<br />
imap.nethere.net<br />
imap.znet.net<br />
imap.simplyweb.net<br />
<br />
Outbound Mail server:<br />
<br />
mail.nethere.net<br />
mail.znet.net<br />
mail.simplyweb.net<br />
<br />
Webmail:<br />
<br />
webmail.nethere.net<br />
webmail.znet.net<br />
webmail.simplyweb.net<br />
<br />
Control Panel:<br />
<br />
cp.nethere.net<br />
cp.znet.net<br />
cp.simplyweb.net<br />
<br />
==== nsrbl Disk Quota Warning ====<br />
<br />
In the event you get a nagios regarding space on the two nsrbl servers, <br />
it is most likely the result of the dccd databases getting too large. <br />
To clear out space, log into each nsrbl server and execute the following:<br />
<br />
df -h<br />
<br />
cd /var/dcc/libexec<br />
<br />
./stop-dccd<br />
<br />
cd /var/dcc<br />
<br />
rm dcc_db*<br />
<br />
sync<br />
<br />
sync<br />
<br />
cd /var/dcc/libexec<br />
<br />
./start-dccd<br />
<br />
ps auxw | grep dccd<br />
<br />
====Update outbound quota for a single ip address====<br />
<br />
We restrict the total number of emails that a customer can send to 2000 email recepients per 24 hour period. This is tracked by the ip address of the computer that connects to the mta server when it sends an outbound emal. The ip address and count is stored in the postfix database server, mta-db.mail.nethere.net. To increase the quota for a single ip address, you can log into either mta-1, mta-2, or console direcly into the database server. To use one of the mta servers:<br />
<br />
<pre><br />
1. Log onto mta-1.mail<br />
2. Switch to mysql<br />
$mysql -h 10.0.0.69 -u policyd_outbound -p<br />
username: policyd_outbound<br />
password: p0stf1x!<br />
3. Review database information<br />
mysql>show databases;<br />
4. Switch to the policyd table. <br />
mysql>use policyd;<br />
5. Review table header information<br />
mysql>show tables;<br />
mysql>describe throttle;<br />
6. Verify the ip address is above quota<br />
mysql>select * from throttle where _from='xxx.xxx.xxx.xxx'; #this is the ip address of the account you are increasing.<br />
7. Increase the maxium number of recepients the ip address can send to<br />
mysql>update throttle SET _rcpt_max = _rcpt_max + 20000 where _from='24.249.205.66';<br />
8. Verify<br />
mysql>select * from throttle where _from='xxx.xxx.xxx.xxx';<br />
</pre><br />
<br />
====Remove a blocked mail server ====<br />
<br />
The message in maillog helps identify where it is blocked.<br />
<br />
<pre><br />
Relaying denied due to excessive spam admin-2:/dist/files/nhmta/nethere/conf/postfix/common/client_reject<br />
Sender address rejected: Relaying denied due to Spam admin-2:/dist/files/nhmta/nethere/conf/postfix/common/sender_reject<br />
Relaying denied due to SPAM policyd???<br />
Client host rejected: ... listed at zen.spamhaus.org=127.0.0.2 nsrbl-1:/named/rbldns/zones/*.spamhaus.org<br />
status=sent (250 2.7.1 Ok, discarded, id=95886-33 - SPAM) user mail protection in provisioning????<br />
<br />
</pre><br />
<br />
====Remove a mail server from policyd blacklist====<br />
<br />
The ip address of a mail server can be placed on a blacklist for several reasons, to high of rate of inbound email from a single ip address, incorrectly formatted helo, etc. In the event you need to remove a mail server's ip from the policyd blacklist, you can log into either mta server of connect directly to the policyd database server, mta-db.mail.nethere.net. To remove the ip address using a mta server:<br />
<br />
<pre><br />
1. Log onto mta-1.mail<br />
2. Switch to mysql<br />
$mysql -h 10.0.0.69 -u policyd_inbound -p policyd<br />
username: policyd_inbound<br />
password: p0stf1x!<br />
3. Display database information<br />
mysql>show databases;<br />
4. Use policyd database<br />
mysql>use policyd;<br />
4. Display table information<br />
mysql>show tables;<br />
mysql>describe blacklist;<br />
5. Verify ip address has been blacklisted<br />
mysql>select * from blacklist where _blacklist='xxx.xxx.xxx.xxx'; <br />
6. Delete the entry<br />
mysql>delete from blacklist where _blacklist='xxx.xxx.xxx.xxx';<br />
7. Verify<br />
mysql> select * from blacklist where _blacklist='xxx.xxx.xxx.xxx';<br />
</pre><br />
<br />
The suspected mail server may also be in the helo table.<br />
<br />
<pre><br />
mysql> describe helo ;<br />
1. Check the IP address<br />
mysql> select * from helo where _host='xxx.xxx.xxx.xxx' ;<br />
2. Check the server helo name <br />
mysql> select * from helo where _helo like '%server%' ;<br />
3. Delete the entry<br />
mysql> delete from helo where _host='xxx.xxx.xxx.xxx' ;<br />
mysql> delete from helo where _helo like '%server%' ;<br />
</pre><br />
<br />
====Remove a mail server from spamhaus blacklist====<br />
<br />
ssh to nsrbl-1.mail and nsrbl-2.mail<br />
<br />
<pre><br />
<br />
cd /named/rbldns/zones <br />
<br />
find which file the IP Address is in.<br />
<br />
grep <IP Addr> *.spamhaus.org<br />
co -l <xxx.spamhaus.org><br />
vi xxx.spamhaus.org<br />
ci -u <xxx.spamhaus.org><br />
<br />
</pre><br />
<br />
====Add ip address to client_reject====<br />
<br />
The client_reject list is one of the ways we combat spam. If we receive spam compaints from a mail server or otherwise identitfy a mail server as a source of spam, we can add it to a client_reject list. An email sent from a mail server on the client_reject list is rejected with a notice saying "Relaying denied due to excessive spam". To add an ip address to the list, we use admin-2. Once the list is updated on admin-2, it is pushed out to mta-1 and mta-2. To add an ip address on admin-2:<br />
<br />
<pre><br />
Log into admin-2.nethere.net.<br />
bash-3.00# cd /dist/files/nhmta/nethere/conf/postfix/common<br />
bash-3.00# co -l client_reject<br />
RCS/client_reject,v --> client_reject<br />
bash-3.00# vi client_reject<br />
<br />
-Add the ip address to the bottom of the list in the appropriate format. It you want to reject any email from the mail server, you would use:<br />
xxx.xxx.xxx REJECT Relaying denied due to excessive spam<br />
where xxx.xxx.xxx is the /24 subnet of the mail server the spam originated from. Do not place a period after the last octet. For example, to add a /16 subnet you would use:<br />
xxx.xxx REJECT Relaying denied due to excessive spam<br />
The REJECT key word tells postfix to reject the email. If you want to allow email from a /24 subnet, use the following format:<br />
xxx.xxx.xxx OK<br />
This tells postfix to accept email from that subnet.<br />
<br />
bash-3.00# ci -u client_reject<br />
RCS/client_reject,v <-- client_reject<br />
new revision: 1.286; previous revision: 1.285<br />
enter log message, terminated with single '.' or end of file:<br />
>> .<br />
done<br />
bash-3.00# make<br />
/nethere/software/nhmta/postfix-2.4.5/sbin/postmap -C . client_reject<br />
mv client_reject.db maps/hash/client_reject.db<br />
bash-3.00# cd /dist/rdist<br />
bash-3.00# gmake nhmta-update<br />
updating host mta-1.mail.nethere.net<br />
updating: /dist/files/nhmta//nethere/conf/postfix/common/maps/hash/client_reject.db<br />
updating: /dist/files/nhmta//nethere/conf/postfix/common/client_reject<br />
updating host mta-2.mail.nethere.net<br />
updating: /dist/files/nhmta//nethere/conf/postfix/common/maps/hash/client_reject.db<br />
updating: /dist/files/nhmta//nethere/conf/postfix/common/client_reject<br />
bash-3.00#<br />
</pre><br />
<br />
====Free mysql disk space on mta-db.mail.nethere.net====<br />
<br />
In response to nagios alert "[Nagios] PROBLEM alert - mta-db.mail.nethere.net/Disk mysql is WARNING", to clear space within the mysql database:<br />
<br />
1. ssh to mta-db.mail.nethere.net.<br />
<br />
2. Log into mysql and execute the following:<br />
<br />
[root@mta-db user]# mysql -uroot -p<br />
<br />
Enter password: dB@dm1N!<br />
<br />
mysql> show databases;<br />
<br />
mysql> use policyd;<br />
<br />
mysql> show tables;<br />
<br />
mysql> describe helo;<br />
<br />
mysql> describe throttle;<br />
<br />
mysql> describe throttle_from_instance;<br />
<br />
mysql> select * from throttle_from_instance limit 10;<br />
<br />
mysql> select now();<br />
<br />
mysql> select unix_timestamp(now());<br />
<br />
mysql> select count(*) from throttle_from_instance where _expire > 1426534697;<br />
<br />
mysql> select max(_expire) from throttle_from_instance;<br />
<br />
mysql> truncate table throttle_from_instance;<br />
<br />
mysql> select count(*) from throttle_from_instance;<br />
<br />
mysql> select * from throttle_from_instance;<br />
<br />
mysql> show tables;<br />
<br />
mysql> optimize table throttle_from_instance;<br />
<br />
mysql> show databases;<br />
<br />
mysql> use policyd;<br />
<br />
mysql> quit<br />
<br />
[root@mta-db user]# df -h<br />
<br />
== Errors ==<br />
<br />
<br />
=== Common webmail errors ===<br />
<br />
<br />
Error:<br />
--cut--<br />
Fatal error: Call to undefined function: applicatio€”¬p() in<br />
/www/webmail.nethere.net/htdocs/x/m/templates/message/navbar.inc on line 7<br />
--cut--<br />
<br />
Solution: Restart apache on webmail-1 (apachectl restart)<br />
<br />
<br />
Error:<br />
--cut--<br />
ERROR There was an error sending your message: unable to add recipient<br />
[webhosting@nethere.com]: Invalid response code received from server<br />
--cut--<br />
<br />
Solution: More than likely DNS/domain issue, but check mta-1.nethere.net to be sure:<br />
--cut--<br />
Aug 30 15:08:54 mta-1 nh/smtpd[50501]: NOQUEUE: reject: RCPT from webmail-1.mail.nethere.net[66.63.128.181]: 450 <orders@mulligrins.com>: Sender address rejected: Domain not found; from=<orders@mulligrins.com> to=<webhosting@nethere.com> proto=ESMTP helo=<webmail.nethere.net><br />
--cut--<br />
Problem: Webmail shows "1-5" messages but inbox appears empty. POP shows messages #1,2,3,4,5... could not be retrieved... server response error cannot open the message file it's gone.<br />
<br />
Solution: Ownership issue<br />
<br />
1) Use the provisioning tool to determine mailbox location:<br />
a) Click on email account, Engineering<br />
b) Location is the "Storage Path:" <br />
<br />
2) Log in to either mail server then change ownership of maildir to mailuser:mailuser<br />
# chown -R mailuser:mailuser /nfs/#/platform/u/s/username<br />
<br />
<br />
<br />
== DEPRECIATED INFORMATION ==<br />
<br />
<br />
<br />
=== Webservers ===<br />
<br />
<br />
<br />
==== Allowing SSI (Server Side Includes) ====<br />
<br />
***** NOTE: This has been DEPRECIATED, all servers now allow SSI by default *****<br />
<br />
We do not allow EXEC permissions for SSI.<br />
<br />
1) Check out Apache config for www.domain.tld in /nethere/conf/apache/conf/<vhosts,nvhosts><br />
# co -l www.domain.tld(.common) <br />
<br />
2) Add the following lines under the <Directory> directive<br />
--cut--<br />
AddType text/html .shtml<br />
AddHandler server-parsed .shtml<br />
--cut--<br />
<br />
3) Edit the "Options" line to be the following:<br />
--cut--<br />
Options Indexes FollowSymLinks IncludesNOEXEC<br />
--cut--<br />
<br />
4) Add a DirectoryIndex line above the <Directory> directive with the following:<br />
--cut--<br />
DirectoryIndex index.shtml index.html index.htm home.html home.shtml index.php<br />
--cut--<br />
<br />
** In summary, the config should look something similar to this **<br />
--cut--<br />
DocumentRoot "/www/www.domain.tld/htdocs"<br />
DirectoryIndex index.shtml index.html index.htm home.html home.shtml <br />
<Directory "/www/www.domain.tld/htdocs"><br />
AddType text/html .shtml<br />
AddHandler server-parsed .shtml<br />
Options Indexes FollowSymLinks IncludesNOEXEC<br />
Order allow,deny<br />
Allow from all<br />
</Directory><br />
--cut--<br />
<br />
5) Check in Apache config<br />
# ci -u www.domain.tld(.common) < /dev/null <br />
<br />
6) Restart Apache<br />
# apachectl configtest<br />
# apachectl restart<br />
<br />
More SSI notes can be found here:<br />
http://httpd.apache.org/docs/1.3/howto/ssi.html<br />
<br />
= SimplyInternet =<br />
<br />
= Z-net =<br />
<br />
= Migrating to LAMP Host Servers =<br />
<br />
Below are outlines of the steps to migrate website & email hosting from nethere to lamphost configured servers. This is almost certainly incomplete at this stage. The exact steps to follow can vary depending on specifics of the client, so use this as a guide only. Be ready to adjust commands or add steps as needed.<br />
<br />
== Edit Nethere DNS ==<br />
<br />
ssh user@admin-1.nethere.net<br />
<br />
sudo -i<br />
<br />
ssh ns1.nethere.net<br />
<br />
cd /named/master<br />
<br />
co -l example.com<br />
<br />
vi example.com<br />
<br />
ci -u example.com<br />
<br />
rndc reload example.com<br />
<br />
Make sure to increment serial number by 1 each time a change is made.<br />
<br />
== Migrate Website to nh3.jcihosting.net ==<br />
<br />
Set TTL to 300 on ns1.nethere.net<br />
<br />
<br />
Open up these web admins to get current account/site info:<br />
<br />
http://toolbox.nethere.net/ -> Provisioning Tool (search for domain name)<br />
<br />
https://secure.johncompanies.com/mgmt/index.html (search for col#####)<br />
<br />
<br />
1. Create the lamphost user.<br />
<br />
https://www.lamphost.com/admin/user/user/create<br />
<br />
username: col#####<br />
<br />
email: same as listed for col##### account<br />
<br />
password: random, secure, won't be shared with client<br />
<br />
defaults on rest of page<br />
<br />
<br />
2. Create the customer account<br />
<br />
https://nh1.jcihosting.net:8443/<br />
<br />
https://nh2.jcihosting.net:8443/<br />
<br />
https://nh3.jcihosting.net:8443/<br />
<br />
https://nh4.jcihosting.net:8443/<br />
<br />
LAMP Host Users -> Add User -> col#####<br />
<br />
<br />
Virtual Hosts -> Add Virtual Host<br />
<br />
Username: Select same username as above from pulldown<br />
<br />
Domain: example.com<br />
<br />
Hosting Plan: select plan that matches what's listed in https://secure.johncompanies.com/mgmt/view.html?cid=col#####<br />
<br />
Add default DNS: No<br />
<br />
<br />
3. Virtual Hosts -> Site Manager (next to domain added)<br />
<br />
Site Manager Users -> Add User<br />
<br />
Username: can be anything - I typically use Optigold Login from nethere admin<br />
<br />
Password: make up something secure and record - this one will go to the user<br />
<br />
<br />
4. Email & FTP -> Add Account<br />
<br />
Email Account: ftp<br />
<br />
Password: make up something secure and record - this one will go to the user<br />
<br />
Enable FTP Access: Yes<br />
<br />
<br />
5. Optional: Add MySQL database if site needs it (WordPress etc)<br />
<br />
<br />
ssh to nh3.jcihosting.net and run 'add_mysql.php <example.com>'<br />
<br />
MySQL database info will be emailed to webmaster@lamphost.com (and output on command line - ignore instructions to run additional commands)<br />
<br />
https://mail.lamphost.com<br />
Sm77DdnQ<br />
<br />
6. Copy over website<br />
<br />
ssh nh3<br />
<br />
cd /var/www/example.com/<br />
<br />
rsync -v --archive --one-file-system --delete --delete-during --rsh=/usr/bin/ssh dsmith@unixweb-10.nethere.net:/www/www.example.com/htdocs ./<br />
<br />
Change 'matt' to your username. Note that I had to add my username to groups httpd & webuser to allow me to copy website files without being root.<br />
<br />
chown -R col#####:col##### htdocs<br />
<br />
<br />
7. If site has MySQL data find the connection info. For example, WP sites will have the info in htdocs/wp-config.php<br />
<br />
mysqldump -h mysqldb-1.webhost.nethere.net -u username --password=password databasename > databse.sql<br />
<br />
mysql exampledotcom < database.sql<br />
<br />
rm database.sql<br />
<br />
<br />
8. If site is ready to be made live on nh3.lamphost.net then update DNS on ns1.nethere.com<br />
<br />
Send email to client with details. Search support@jcihosting.com Sent folder for emails with Subject "updated hosting for" for example emails. You will have to adjust as needed (username/passwords, different info based on specific client).<br />
<br />
9. After you finish, update the johncompanies.com backend for the col0# CID and change the nethere hosting server to nh3.jcihosting.net.<br />
<br />
10. To add a SSL certificate<br />
letsencrypt-create.php <domain.com><br />
<br />
To remove a SSL certificate<br />
del-letsencrypt-certificate.php <domain.com><br />
<br />
== Migrate Email to nh3.jcihosting.net ==<br />
<br />
If migrating website & emails both follow the above steps to transfer the website first.<br />
<br />
If migrating emails only the follow the first steps 1-3 of the above steps to set up the virtualhost on nh3.<br />
<br />
<br />
1. Set TTL for MX record to 300.<br />
<br />
<br />
2. Generate email report:<br />
<br />
Log in to http://toolbox.nethere.net/<br />
<br />
Select Provisioning Tool and search for the domain<br />
<br />
Navigate to the Email section and then Generate Report.<br />
<br />
Select these fields for the report:<br />
<br />
Mail address<br />
<br />
Alias<br />
<br />
Forwarding address<br />
<br />
Mail storage path<br />
<br />
Copy the result of the report and save it as a text file in /tmp/ on nh3.jcihosting.net named example.com.txt (replace domain name)<br />
<br />
<br />
3. Create the e-mail accounts<br />
<br />
Make sure the virtualhost has enough email accounts allocated before proceeding. <br />
Edit the virtualhosts.virtualhost_settings table directly if need be to increase allocation.<br />
<br />
Log in here: https://nh3.jcihosting.net:8443/<br />
Select "MySQL.LH"<br />
Go to the "virtualhosts" database<br />
Search the "virtualhost_settings" settings for 'example.com' <br />
and adjust the 'email_accounts' setting.<br />
<br />
<br />
Run this script once and only once:<br />
<pre><br />
/usr/local/bin/migrate_mail_accounts_from_nethere.php example.com<br />
</pre><br />
<br />
That will create all email accounts and aliases.<br />
<br />
A file containing the list of email accounts and their new passwords will <br />
be created as /tmp/example.com-passwords.txt. Grab a copy of that for <br />
sending to the client.<br />
<br />
<br />
4. Sync the emails<br />
<br />
Another file will be created that contains the bash commands to sync emails from the old server. This will be named /tmp/example.com-sync.sh<br />
<br />
Copy the file somewhere like the /root/ home dir and add "#!/bin/bash" as the first line and change perms to 750. Now you can run this script to sync all mails from nethere to nh3. Run it immediately to get an initial sync of emails.<br />
<br />
When ready to switch email hosting over run the sync script one more time, then update DNS on nh3.nethere.net.<br />
<br />
If you'd like to sync again after the move (if there was a delay or you think there may have been incoming emails during the transtion) remove the '--del' option from the sync script and then re-run.<br />
<br />
<br />
5. Send email to customer<br />
<br />
Search support@jcihosting.com Sent folder for messages with Subject "new email hosting for" for examples of emails to use for communicating with the client.<br />
<br />
6. Update DNS and put NetHere mail on hold<br />
Once DNS has been pointed to the new server, go into the NetHere Toolbox http://toolbox.nethere.net/ and go to Domain -> General and then set Status to 'On Hold'.<br />
<br />
7. Update host in JohnCompanies database.<br />
<br />
== Moving between Host Servers on Lamphost ==<br />
<br />
You should delete the old virtual host before creating the new<br />
virtual host. It may take up to an hour to delete the old virtual<br />
host. To speed up the delete:<br />
<pre><br />
/opt/lamphost/cron-bin/del_virtualhosts.php<br />
</pre><br />
<br />
If you create the new virtual host before, you will need to update<br />
the mylampsite.com:<br />
<pre><br />
/opt/lamphost/bin/update_mylampsite_subdomain.php databaseproviders.com<br />
</pre><br />
<br />
== Upgrade a Lamphost server to increase database limit ==<br />
<br />
I've increased the database limit for this customer to 2. Now they can add a second db via the Site Manager admin.<br />
<br />
<pre><br />
Here is how I made the adjustment (we don't have a friendly admin for this function):<br />
Log in to Server Manager: https://nh3.jcihosting.net:8443/<br />
Select the "MySQL.LH" phpMyAdmin<br />
Select 'virtualhosts' db.<br />
Adjust the 'databases' setting for the argee.com domain in the 'virtualhost_settings' table.<br />
</pre><br />
<br />
== looking in the mail log files ==<br />
<br />
exigrep</div>75.80.128.147